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1.  Introduction 


This  research  program  addressed  an  unmet  need  in  the  training  for  treatment  of  head  trauma,  focusing 
on  ocular,  peri-ocular  and  facial  trauma.  While  improvements  in  armor  systems  have  reduced  injuries  to 
the  trunk  and  extremities,  the  face  and  head  remain  relatively  vulnerable.  Opportunities  for  medics  to 
learn  the  treatment  of  facial  and  head  trauma  are  limited  due  to  the  lack  of  realistic  training  simulators 
for  this  area  of  the  body.  For  military  ophthalmologists,  critical  trauma  skills  such  as  microsurgical 
suturing  are  no  longer  routinely  taught  in  US  residencies.  Consequently,  the  burden  of  developing  and 
maintaining  these  essential  trauma  skills  is  on  the  ophthalmologists  themselves.  We  addressed  these 
dual  needs  by  creating  a  new  prototype  training  system  to  simulate  injuries  of  the  orbit  and  peri-orbital 
tissues,  including  elements  addressing  the  mandible,  maxilla,  and  upper  airway.  The  system  functions  as 
a  procedural  training  system  and  was  also  designed  for  integration  into  a  mannequin-based  trauma 
simulator  to  support  advanced,  multiple  trauma  scenarios.  It  was  designed  to  incorporate  performance 
measurement  sensors  and  algorithms  together  with  physical  mannequin  components,  conventional 
displays  and  augmented  presentations  of  teaching  content,  and  to  provide  meaningful  feedback  to 
enhance  learning.  The  research  was  performed  in  three  overlapping  phases:  improvement  of  an  earlier 
proof-of-principle  eye  trauma  system  for  preliminary  user  testing;  substantial  revision  and  expansion  of 
the  eye  trauma  system  to  form  a  full  face/neck  trauma  system  with  additional  eye  trauma  and  face 
trauma  components;  and  development  and  integration  of  a  mannequin-compatible  version  of  a  selected 
set  of  relevant  components  to  enable  more  comprehensive  trauma  scenarios. 

This  program  was  originally  conceived  as  a  two  year  research  and  development  effort;  no  cost 
extensions  supported  the  continuing  work  through  an  additional  18  months,  accounting  for  initial  delays 
and  personnel  changes  during  the  first  two  years. 

This  final  report  summarizes  the  accomplishments  of  the  entire  program. 


2.  Body 

2.1.  Project  background:  proof-of-principle  simulator  development 

The  origin  of  this  research  program  was  a  proof-of-principle  system  developed  in  our  laboratory  in 
response  to  training  needs  expressed  by  our  subject  matter  expert,  Dr.  Robert  Mazzoli,  M.D.  now  of  the 
Vision  Center  of  Excellence.  In  that  earlier  research,  Dr.  Paul  Neumann,  Ph.D.  and  colleagues  created  a 
physical  silicone  head  form  with  a  simple  replaceable  left  eye  module,  and  coupled  it  to  a  pair  of 
magnetic  position  tracking  sensors  attached  with  Velcro  straps  to  the  surgeon's  index  fingers.  In 
performing  lid  laceration  repair,  the  index  figure  motions  were  measured,  with  performance  statistics 
previously  employed  in  a  laparoscopy  trainer  that  we  developed,  including  total  exercise  duration, 
instrument  path  length,  an  indication  of  motion  smoothness,  and  a  measure  of  amount  of  hand 
reorientation.  The  original  project  included  demonstrations  for  the  subject  matter  experts  and  others, 
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but  did  not  extend  to  the  level  of  collecting  expert  and  novice  performance  data  to  create  a  means  for 
distinguishing  the  groups  and  determining  when  a  novice  reached  some  threshold  of  acceptable 
performance. 


Figure  1.  Proof-of-principle  physical  mannequin,  motion  tracking  system,  sample  results  plot,  circa  2010. 

While  a  promising  beginning,  numerous  desirable  changes  were  identified  by  Dr.  Mazzoli  and  others. 
Anatomically,  the  eye  lid  consisted  of  only  one  layer  of  silicone;  real  lids  have  two  major  layers  of 
differing  biomechanical  character  and  additional  structures  and  landmarks  important  to  correct  surgical 
repair.  A  hollow  eye  globe  was  filled  with  a  simulated  vitreous  humor  substitute,  however  the  cornea 
was  opaque  and  the  anterior  chamber  was  a  solid  structure;  for  realism  and  to  allow  simulated  injury  of 
the  globe,  a  transparent,  thin  cornea  is  necessary,  together  with  an  iris/pupil,  which  ideally  should 
indicate  light  response  or  pathology.  The  lid  lacerations  were  manually  created,  with  limitations  on  how 
repeatable  the  simulated  injury  could  be  generated;  a  training  system  with  standardized,  replicable 
scenarios  would  be  more  useful  and  reliable  as  a  measurement/testing  system.  The  tracking  sensors 
were  attached  to  the  user's  fingers,  providing  at  best  a  poor  approximation  of  the  motions  of  the 
instrument  tips;  revision  of  this  and  the  expansion  of  the  instrument  suite  were  requested.  Lastly,  while 
the  laparoscopy  simulator-based  scoring  system  used  has  been  shown  to  be  able  to  distinguish  expert 
from  novice  laparoscopic  surgeons  (on  a  suitable  simulator),  it  is  a  post-performance  score  and  cannot 
easily  be  used  to  make  detailed  recommendations  for  improvement  -  a  more  detailed  evaluation  and 
feedback  system  is  desirable. 

In  addition  to  addressing  these  system  improvements,  the  original  vision  was  to  create  a  larger  suite  of 
scenarios,  to  be  supported  by  a  series  of  replaceable  eye  trauma  modules:  globe  laceration  (scleral 
and/or  corneal),  more  complex  lid  lacerations,  and  retrobulbar  hemorrhage. 


The  current  research  also  extends  the  simulator  to  create  a  system  that  supports  training  of  first 
responders  in  management  of  eye  trauma  as  well.  As  ocular  injuries  are  typically  addressed  surgically 
only  after  life  threatening  conditions  are  managed,  it  is  essential  that  first  responders  properly  protect 
the  injured  eye  prior  to  transport,  avoid  causing  additional  damage  (typically  by  applying  inappropriate 
pressure),  and  in  the  case  of  retrobulbar  hemorrhage,  quickly  identify  the  condition  and  take 
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appropriate  steps  to  relieve  pressure  on  the  optic  nerve  to  avoid  permanent  loss  of  vision.  For  this 
reason,  developing  capabilities  to  present  eye  injuries  to  medics  and  detect  interventions  was  deemed 
to  be  important  for  ongoing  development.  Because  eye  trauma  is  often  accompanied  by  face  trauma, 
including  additional  facial  trauma  elements  and  airway  management  features  were  also  considered, 
together  with  integration  of  a  revised/improved  simulator  with  a  full-body  mannequin-based  simulator 
for  combined  head/body  trauma  capabilities. 

These  needs  lead  to  the  current  research  effort,  which  seeks  to  address  them  in  three  overlapping 
phases: 


•  Revision  and  evaluation  of  the  proof-of-principle  eye  trauma  system,  including  user  testing 

•  Expansion  of  the  eye  trauma  system  to  include  additional  ocular  elements,  facial  trauma 
components  and  upper  airway  management  elements 

•  Integration  of  the  procedural  trainer  components  into  a  mannequin  system 

Over  the  course  of  these  phases,  we  created  a  substantially  more  advanced  prototype  that  includes 
many  of  the  desired  features  and  developed  proof-of-concept  models  to  examine  the  rest. 

For  the  evaluation  and  testing  elements  of  this  project,  we  have  brought  the  system  to  the  annual 
Ocular  Trauma  course  offered  at  USUHS  to  obtain  feedback  and  direction  for  ongoing  development  as 
well  as  to  help  in  the  training  of  ophthalmology  residents 

We  have  further  been  involved  in  a  simulator  validation  program  associated  with  the  Ocular  Trauma 
course,  representing  a  near-commercial  level  simulator  to  be  used  in  comparison  with  the  current 
standard  of  education,  namely  live  animal  and  animal-tissue-based  training. 

Based  on  these  developments,  we  applied  for  and  were  awarded  additional  funding  to  create  additional 
features  for  irreparable  globe  injuries,  facial  fracture  response  training  and  modification  of  the  system 
for  compatibility  with  an  additional  full-body  mannequin  system. 

The  accomplishments,  described  by  the  various  milestones  related  to  these  phases,  are  detailed  in  the 
sections  below. 

2.2.  Quarterly  Report  Administrative  Details 

The  prior  quarterly  report  covered  the  period  up  to  and  including  15  December  2014.  The  following 
details  cover  the  remainder  of  the  program,  up  to  the  end  date  of  10  February  2015. 

1.  Contract  No.:  _  W81XWH-11-C-0095 _ 

2.  Report  Date:  3/28/2016 _ 

3.  Reporting  period  from _ 16  December  2014 _ to _ 10  February  2015 _ 

4.  PI:  _Mark  P.  Ottensmeyer,  Ph.D. _ 
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5. 


Telephone  No.:  617-768-8783 


6.  Institution: _ Massachusetts  General  Hospital _ 

7.  Project  Title:_Development  of  an  Ocular  and  Craniofacial  Trauma  Treatment  Training  System 

8.  Current  staff,  with  percent  effort  of  each  on  project  (applies  to  final  quarter  prior  to  final  month 
for  reporting): 

Mark  P.  Ottensmeyer,  Ph.D.  P.l.  35% 

Gianluca  De  Novi,  Ph.D.  Computer  Scientist,  Post-Doc  35% 

Gregory  J.  Loan  Clinical  Engineer,  Fabrication  Lead  35% 

Steven  L.  Dawson,  M.D.  co-l,  SME  0%* 

Maria  J.  Troulis,  D.D.S.  SME  OX1 


9.  Contract  expenditures  to  date  (as  applicable): 


11/30/14-02/10/15 

/ 

Cumulative 

Personnel: 

$  16,852.60 

/ 

$  1,140,881.80 

Fringe  Benefits: 

6176.84 

/ 

409,544.33 

Supplies: 

554.32 

/ 

71,577.24 

Travel: 

2866.04 

/ 

35,997.06 

Equipment: 

0.00 

/ 

139,858.50 

Non-capital  equipment: 

-430.322 

/ 

28,342.08 

Consultants: 

0.00 

/ 

15,346.40 

Other: 

-9896.183 

/ 

81,086.57 

Subtotal: 

$16,123.30 

/ 

$1,922,633.98 

Indirect  Costs: 

11,931.27 

/ 

1,329,335.02 

Fee: 

0.00 

/ 

0.00 

Total: 

$28,054.57 

/ 

$3,251,969.00 

Budget 

3,251,969.00 

1  Dr.  Dawson  and  Dr.  Troulis  continued  to  serve  as  SMEs  and  advisors  during  the  period  of  the  no-cost  extension, 
with  no  further  salary  component  requested. 

2  Return/refund  of  incompatible  graphics  card  for  primary  software  development  workstation  purchased  in  prior 
quarter 

3  Final  journal  entry  adjustments  to  correct  for  differences  between  financial  reporting  intervals  and  contract  end 
date. 


9 


10.  Comments  on  administrative  and  logistical  issues. 

This  document  summarizes  the  full  research  program  that  lead  to  the  development  of  the  Ocular- 
Craniofacial  Trauma  Training  Simulator,  and  goes  into  additional  detail  for  the  period  following  the 
penultimate  quarterly  report. 

This  final  report  was  completed  and  submitted  after  the  specified  deadline,  due  primarily  to  issues 
related  to  first  time  parenthood  of  the  PI  coincident  with  the  no-cost  extension  (NCE)  period  and 
underestimation  of  the  requirements  of  parallel  grants  that  began  as  this  program  reached  its 
conclusion.  The  PI  recognizes  the  importance  of  the  submission  of  the  final  report  and  regrets  the 
complications  that  have  resulted  from  the  delinquency  of  this  document. 

Personnel: 

During  the  final  reporting  period,  our  staff  remained  stable,  comprising  Dr.  Ottensmeyer  (PI,  mechanical 
engineering),  Dr.  Gianluca  De  Novi  (software  architecture)  and  Mr.  Gregory  Loan  (anatomical  modeling, 
fabrication). 

Review  of  significant  non-personnel  expenses  during  final  reporting  period 

During  the  final  reporting  period,  the  following  significant  expenses  were  incurred: 

Equipment/Supplies: 

•  None.  Only  minor  expenses  were  incurred,  related  to  preparation  of  the  simulator  for 
demonstration  at  the  2015  International  Meeting  on  Simulation  in  Healthcare  conference  in 
New  Orleans,  LA,  10-14  January,  2015 


Travel: 


•  Dr.  Ottensmeyer  and  Dr.  De  Novi  presented  the  simulator  at  the  2015  International  Meeting  on 
Simulation  in  Healthcare  conference  in  New  Orleans,  LA,  10-14  January,  2015  in  the  context  of 
the  TATRC  demo  area. 

IRB/Human  use  protocol  status: 

There  was  no  IRB/Human  Use  activity  during  the  final  months  of  the  program.  Following  the  end  of  this 
research  effort,  the  protocol  that  had  been  approved  by  the  IRB  at  Massachusetts  General  Hospital  was 
extended  to  enable  continuing  use  of  the  simulator  at  the  annual  USUHS  Ocular  Trauma  Course.  The 
continuing  review  documentation  was  forwarded  to  HRPO  with  information  regarding  the  end  of  the 
program.  They  determined  that  no  further  review  would  be  necessary  by  HRPO  in  relation  to  this 
program. 

SME4  and  other  expert  interactions: 


4  Subject  Matter  Expert 


10 


On  15  December  2014,  Mr.  John  James  of  Sawbones  (Pacific  Research  Laboratories)  visited  the 
Simulation  Group  facility  to  review  the  Ocular  Trauma  simulator  to  evaluate  whether  the  system  or 
components  thereof  would  be  of  interest  to  his  company.  No  immediate  efforts  towards  licensing 
elements  of  the  system  were  pursued,  however  we  will  continue  to  maintain  contact  and  explore  future 
options  as  the  project  continues  with  additional  support  (see  below). 

Following  our  presentation  at  the  TATRC  "Corral"  at  the  2015  IMSH  conference,  we  were  contacted  by 
the  Center  for  Learning  Clinical  Attitudes  and  Skills  (Centre  d'Apprentissage  des  Attitudes  et  Habilites 
Cliniques),  a  CAE  Healthcare-affiliated  Simulation  Center  in  Montreal,  Canada,  to  inquire  regarding  the 
cost  of  acquiring  one  of  the  simulators.  We  have  replied  to  their  medical  director  and  await  a  further 
response. 

Lastly,  also  in  relation  to  the  presentation  at  IMSH,  we  made  contact  with  Markus  Schill  of  VR  Magic, 
makers  of  the  Eyesi  Cataract  and  Vitreoretinal  VR-based  simulators.  We  have  scheduled  a  conference 
call  for  shortly  after  the  end  of  this  program  to  discuss  application  of  our  gesture-recognition  algorithms 
and  our  interchangeable  instrument  designs  to  their  system. 

Funding  Proposals 

There  were  no  new  proposals  for  funding  initiated  during  this  final  reporting  interval. 

A  grant  from  the  Center  for  Integration  of  Medicine  and  Innovative  Technology  (CIMIT)  awaits  the  arrival 
of  our  new  post-doctoral  research  fellow  to  initiate  an  effort  to  develop  an  enucleation  component, 
enhanced  mid-facial  fracture  components,  and  modification  of  the  head  form  to  be  compatible  with 
other  mannequin-based  simulators. 

The  USUHS  Ocular  Trauma  course  is  involved  in  a  Simulator  Validation  Program  effort  in  which  our 
simulator  is  being  compared  with  the  standard  animal-tissue  based  training.  Expenses  to  transport  and 
operate  our  system  and  for  the  consumable  facial  trauma  modules  at  the  2015  Course  will  be 
reimbursed  through  the  SVP  program. 

As  the  period  of  performance  ended,  we  had  scheduled  discussions  with  SimQuest  Solutions,  Inc.  to 
discuss  whether  our  physical  models  and  sensorized  canthotomy/cantholysis  components  might  be 
suitable  to  support  an  SBIR  application  by  SimQuest,  (DHP15-001,  "Lateral  Canthotomy  and  Cantholysis 
Training  System")  and  will  pursue  the  discussions  further. 


2.3.  Research  Accomplishments  by  Milestone 

For  each  of  the  individual  milestone  tasks,  the  accomplishments  over  the  whole  of  the  program  will  be 
summarized,  with  final  elements  of  the  effort  highlighted  in  additional  detail  where  relevant. 

The  "Month  #"  reference  follows  the  language  of  the  contract,  although  progress  proceeded  in  parallel 
across  multiple  milestones  and  completion  times  of  the  milestones  varied  from  that  of  the  original 
calendar,  in  part  due  to  extension  of  the  program  during  the  periods  of  the  no-cost  extensions. 
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In  the  proposal,  certain  separate  milestones  were  listed  for  completion  in  the  same  month,  but  were 
combined  in  the  contract  language.  In  this  report,  such  items  are  separated  as  parts  a,  b,  etc.  where 
relevant. 

2.3.1.  “Month  1”  -  Review  of  craniofacial  trauma  training  needs  with  Subject  Matter  Experts 
(SMEs),  creation  of  specifications,  limitations,  detailed  analysis  for  trauma  cases. 

This  milestone,  relevant  to  the  Phase  I  work,  was  completed  in  year  1.  This  milestone  was  accomplished 
through  multiple  discussions  with  our  SMEs,  their  colleagues,  contacts  established  through  TATRC  and  at 
the  2011  AFSIM  meeting,  and  review  of  multiple  literature  sources  including  standard  military  medical 
training  and  civilian  eye  trauma  references.  Personnel  consulted  included  Dr.  Robert  Mazzoli  (SME), 
and  his  colleagues  at  MAMC,  Dr.  (LTC)  Adam  Buchanan  and  Dr.  (MAJ)  Jeffrey  Shere;  Dr.  Jeffrey  Cain;  Dr. 
(COL)  Robert  Hale  and  his  colleage  Dr.  Anthony  Johnson  at  AISR;  and  surgeons  at  the  Massachusetts  Eye 
and  Ear  Infirmary  (MEEI)  Drs.  Suzanne  Freitag,  Justin  Kanoff  and  Daniel  Lefebvre.  In  addition  to 
discussions,  we  were  able  to  closely  observe  the  MEEI  surgeons  perform  multiple  ocular  procedures, 
including  lid  laceration  repair,  suborbital  fracture  repair,  enucleation  (a  procedure  also  mentioned  by  Dr. 
Hale),  and  a  procedure  to  remove  a  dislodged  artificial  lens.  Dr.  Maria  Troulis,  one  of  our  MGH  SMEs,  is 
a  maxillofacial  reconstructive  surgeon;  with  her  we  reviewed  photographs  and  CT  reconstructions  of 
examples  of  severe  facial  trauma  resulting  from  multiple  causes,  as  well  as  typical  classes  of  upper  and 
lower  facial  fractures  (e.g.  Le  Fort  fractures,  typical  jaw  fractures). 

As  a  whole,  no  changes  were  made  to  the  plans  for  the  types  of  injuries  simulated  by  the  system. 
Corresponding  with  preliminary  discussions  that  lead  to  the  original  project  scope,  the  desired  scope  of 
injury  scenarios  that  the  system  address  included: 

For  surgical  scenarios: 

•  Full  thickness  lid  lacerations 

•  Lid  laceration  with  canalicular  damage 

•  Intact  globe  with  retrobulbar  hemorrhage 

•  Globe  laceration  (scleral  and  corneal) 

•  Combinations  of  the  above 

For  first  responders: 

•  Protection  of  the  eye  using  an  eye  shield  and  stabilization  of  penetrating  objects 

•  Intact  globe  with  retrobulbar  hemorrhage 

•  Removal  of  non-penetrating  foreign  bodies 

•  Facial  hemorrhage  control 

•  Management  of  airway  with  severe  damage  to  the  lower  face 

•  Surgical  airway 


12 


Retrobulbar  hemorrhage  appeared  in  both  lists  as  this  is  a  condition  that  requires  recognition  and 
urgent  response  whether  in  the  field  or  at  a  medical  facility,  namely  lateral  canthotomy  and  cantholysis, 
is  included  both  in  surgical  training  and  also  in  the  SOF  Medical  Handbook. 

Our  observations  showed  that  for  oculoplastics  and  facial  procedures,  magnifying  loupes  would  be  worn 
by  physicians,  so  they  would  be  able  to  make  use  of  video  displays  showing  teaching/feedback  content 
placed  near  the  simulator.  For  corneal  and  scleral  procedures,  an  operating  microscope  would  be  in  use 
and  the  physician  would  be  "heads-down"  for  much  a  procedure  -  this  suggested  that  an  augmented 
reality  system  to  provide  content  through  the  microscope  would  be  desirable. 

During  this  period,  we  also  began  to  gather  the  list  of  necessary  and  desirable  instruments  that  should 
be  provided  to  the  trainees  and  which  would  need  to  be  tracked  in  position,  orientation  and 
actuation/force  to  provide  quantitative  data  for  evaluation  and  providing  context-sensitive  feedback. 

Dr.  Cain  and  others  recommended  that  facial  trauma  presented  should  be  either  relatively  minimal,  so 
that  intubation  would  be  the  preferred  choice  for  establishing  an  artificial  airway,  or  so  severe  that 
trainees  would  be  faced  with  no  alternative  by  to  perform  a  cricothyroidotomy. 

While  facial  fractures  were  included  as  part  of  the  damage  calling  for  hemorrhage  control  and 
management  of  complicated  airway,  surgical  repair  and/or  stabilization  of  the  bone  fragments  was 
outside  of  the  scope  of  this  simulation  system  program.  Similarly,  enucleation  was  been  recommended 
as  a  valuable  scenario.  In  Year  3,  a  proposal  for  funding  to  create  modules  for  both  of  these  scenarios 
was  submitted  and  approved  for  funding  by  the  Center  for  Integration  of  Medicine  and  Innovative 
Technology  (CIMIT),  as  one  of  a  series  of  proposals  supporting  the  Joint  Warfighter  Medical  Research 
Program.  Work  on  this  project  will  follow  after  the  completion  of  this  contract. 

Of  the  scenarios  listed  above,  we  created  a  fully-developed  scenario  to  teach  repair  of  full-thickness  lid 
laceration  including  content,  tracking  and  realistic  physical  components.  We  created  an  animatronic 
system  to  generate  proptosis  for  retrobulbar  hemorrhage;  an  electronic  valve  system  that  has  the 
capacity  to  direct  fake  blood  to  multiple  sites;  a  series  of  eye  globe  models  that  support  creation  of 
incisions  in  the  cornea  and  sclera  for  teaching  repair;  facial  trauma  modules  that  can  be  interchanged  to 
show  more  or  less  injury  to  the  face;  a  highly  realistic  oral/nasal/tracheal  airway  that  can  be  intubated; 
an  augmented  reality  simulator  system  that  can  be  used  to  present  both  traditional  graphical  displays 
of  information  as  well  as  projected  imagery  on  the  anatomy  and  graphics  merged  with  a  direct 
microscope  view  of  the  silicone  anatomy;  sensing  systems  to  detect  cuts  through  simulated  tissue  at  the 
lateral  canthus  of  the  eye  and  the  skin  over  the  larynx;  "unlockable"  facial  fracture  components  that  can 
present  palpable  mobility  of  jaw  and  facial  "bone"  structures;  and  a  software  architecture  that 
quantitatively  measures  and  responds  to  the  performance  of  the  trainee,  which  can  be  adapted  to  a 
wide  range  of  procedures.  Lastly,  we  modified  the  procedural  head/neck  form  of  the  system  to  be 
mounted  in  a  commercial  full-body  mannequin  to  support  the  capability  for  use  in  a  multiple 
trauma/first  responder  scenario.  Many  of  these  features  have  been  demonstrated  to  elicit  expert 
opinion,  and  the  lid  laceration  scenario  and  features  have  been  used  for  three  years  at  the  USUHS 
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Ocular  Trauma  course,  including  as  part  of  a  comparison  study  between  training  on  animal  tissue  and  on 
our  silicone-based  simulator,  which  is  ongoing  at  the  conclusion  of  this  phase  of  our  research  program. 

2.3.2.  “Month  2”  - 

2. 3. 2. a.  Review  of  treatment  modes  and  potential  error  modes  to  be  detected 
This  milestone,  relevant  to  the  Phase  I  work,  was  completed  in  Year  1.  Based  on  discussions  with  the 
SMEs,  primarily  Dr.  Mazzoli  and  Dr.  Freitag,  we  developed  a  series  of  measurements  relevant  to  the 
treatment  modes  appropriate  for  the  scenarios  described  above.  These  measurements  have  thresholds 
or  criteria  that  will  allow  for  the  determination  of  success  or  error  states. 

For  the  surgical  interventions,  the  primary  source  of  performance  information  is  the  6  degree  of 
freedom  (DOF)  magnetic  position  trackers  described  in  more  detail  below.  With  instrument  motion 
data,  we  measure  global  criteria  including  path  length,  performance  time,  motion  smoothness,  and 
amount  of  instrument  reorientation  and  can  derive  additional  global  measurements.  While  useful  from 
an  overall  scoring  perspective,  more  detailed  measures  are  needed  to  provide  useful  immediate 
feedback  for  improvement.  As  a  result,  we  determined  that  detection  of  the  sequence  of  surgically 
relevant  gestures  is  an  important  element  of  evaluation  of  treatment  and  errors.  Performing  gestures 
out  of  sequence  or  missing  elements  of  a  procedure  can  be  flagged  as  errors  or  deviations  from  the 
ideal.  Surgical  gesture  recognition  and  analysis  methods  that  we  developed  are  described  in  sections 

2.3.2.b.v,  2.3.5. b,  and  2.3.7.c.iv.  Analysis  of  the  gestures  allows  detection  of  numbers  and  types  of 
suture  throws,  knot  types,  potentially  size  of  needle  "bites/'  and  sequencing  of  these.  As  a  parallel 
measure  of  performance,  multiple  eye  surgeons  mentioned  that  with  increasing  stress  or  also  through 
inexperience,  trainees  may  grasp  instrument  with  excessive  force;  we  are  able  to  measure  this  in  part 
through  sensors  installed  on  the  surgical  instrument  (strain  gages  to  measure  instrument  deformation). 
Feedback  can  be  given  to  the  user  immediately,  allowing  him  or  her  to  improve  performance  during  the 
course  of  a  scenario,  rather  than  receiving  such  feedback  only  after  completion  and  receipt  of  a  final 
score. 

For  general  management  of  the  head,  relative  positioning  of  the  head  can  be  sensed  using  the  magnetic 
tracking  system.  This  is  especially  important  when  cervical  spinal  injury  scenarios  are  considered  and  it 
also  allows  the  system  to  synchronize  the  motion  of  the  head  and  hand  gestures  during  data  acquisition. 

Additional  measurement  and  sensor  systems  that  were  identified  and  studied  to  the  level  of  proof-of- 
concept  but  that  were  not  implemented  in  the  final  prototype  version  of  the  system  include: 

•  Sensing  of  force/pressure  applied  directly  to  the  globe,  indicating  incorrect  application  of  an  eye 
shield 

•  Detection  of  water-tightness  of  globe  suturing  and  establishment  of  hemostasis 

•  Sensing  direct  pressure  applied  to  simulated  facial  wounds/bleeding 

Measurements  identified  for  consideration  in  future  versions  include: 
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•  Automatic  measurement  of  alignment  of  geometric/anatomical  landmarks,  e.g.  lid  margins, 
corneo-scleral  margin 

•  Detection  of  the  presence  of  a  scleral  shield,  placed  on  the  globe  to  protect  it  during  lid  repair, 
but  left  behind  after  completion  of  the  scenario 

The  system  can  accommodate  the  implementation  of  such  features  in  future  developments  should  they 
be  required.  An  electronic  valve  system  includes  components  to  pressurize  the  eye  globes,  the  selected 
microcontroller  system  has  sufficient  analog  and  digital  input  lines  to  sense  multiple  pressure  and  force 
sites,  an  improved/repositioned  video  camera  could  detect  anatomical  alignments,  and  inclusion  of  the 
scleral  shield  insertion  suction  device  among  the  position-tracked  ophthalmological  instrument  suite  will 
allow  for  tracking  the  insertion  and  removal  of  the  shield. 

2.3.2. b.  Implement  improvements  for  upgraded  proof-of-concept  stand-alone  eye  trauma 
simulator  based  on  feedback  from  preliminary  user  testing  by  CIMIT5  project 
advisors. 

This  phase  of  the  work  comprised  the  largest  element  of  the  Phase  I  work,  and  focused  on 
improvements  to  the  simulated  anatomy,  the  suite  of  instruments  available  to  the  surgeon/trainee, 
position  and  additional  sensing  capabilities,  development  of  a  functional,  structural  system  for  the 
simulator,  a  versatile  user  interface  and  data  collection  architecture,  and  development  of  didactic 
content  and  a  presentation  architecture  appropriate  to  the  scenarios  under  development. 

Accomplishments  in  these  areas  are  described  as  chronological  developments  to  provide  context. 
Additional  modifications  to  the  design  based  on  later  phases  of  the  work  are  described  in  later  sections. 

2.3.2. b.i.  Eye  globe  anatomy 

Improvements  to  the  original,  proof-of-principle  system  eye  globe  are  numerous.  The  original  globe  had 
an  opaque,  tinted  cornea,  no  lens  or  iris,  and  a  preliminary  estimate  of  scleral  thickness.  In  year  1  we: 

•  Created  clear,  anatomically  sized  lens;  more  accurate  scleral  thickness  with  integrated  nylon 
mesh  to  resist  silicone  fracturing  and  enable  anchoring  of  sutures;  colored  irises  of  anatomical 
thickness;  transparent  corneas  with  open  (rather  than  solid)  anterior  chamber 

•  Evaluated,  with  assistance  from  our  SMEs,  a  series  of  viscoelastic  materials  for  use  as  the 
vitreous  humor  within  the  globe,  including  polyvinyl  acetate  "slime",  egg  white  (as  suggested 
through  contact  with  Phillips  Studio,  a  potential  commercial  contact  for  production  of 
anatomical  components),  and  a  series  of  silicone  and  water-based  personal  lubricant  materials 

•  Modified  the  globe  design  to  approach  compatibility  with  actively  driven  actuation  of  eye 
motion  and  integration  with  an  internal  pupil  contraction  mechanism,  pressure/force  sensing. 

Further  developments  in  later  years  are  described  below  in  the  context  of  development  of  the  eye 

motion  mechanism,  exploration  of  pupil  contraction  mechanisms  and  development  of  the  full-face 

replaceable  modules. 


5  Center  for  Integration  of  Medicine  and  Innovative  Technology,  Massachusetts  General  Hospital,  Boston,  MA 
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Figure  2:  Top  row:  original  eye  globe  with  opaque  cornea;  new  transparent  lens;  molded,  mesh-reinforced  sclera; 
molding  of  iris;  iris  installed  on  new  globe.  Middle  row:  molded  transparent  corneas;  assembled  globe  with  normal 
sclera  thickness,  iris,  cornea  and  lens;  samples  of  gel/lubricant  materials  compatible  with  silicone  anatomy 
evaluated  for  similarity  to  vitreous  humor.  Bottom  row:  "stringiness"  property  of  Astroglide  shown  as  fingers  are 
separated,  similar  to  vitreous  humor  characteristic;  new  globe  concept  with  rigid  backing  for  use  with  animatronic 
eye  module  (Phase  II). 

We  also  made  contact  with  Phillips  Studio,  UK,  who  creates  commercial  globe  models  for  training  vitreo- 
retinal  surgery,  cataract  surgery  and  strabismus.  They  did  not  have  trauma  simulation  products,  and 
their  globe  models  were  not  directly  compatible  with  our  concepts  for  the  Phase  I  or  Phase  II  systems. 


Figure  3:  Samples  of  vitreo-retinal  surgery  simulation  globes  from  Phillips  Studio. 
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2.3.2.b.ii.  Eyelid  and  periocular  module  anatomy 
The  overall  module  structure  for  the  Phase  I  system  was  retained  during  the  early  portions  of  the  work 
for  compatibility  with  the  original  head  form.  Numerous  modifications  and  improvements  were 
implemented  during  Y1  of  the  program,  however. 

•  A  series  of  experiments  to  evaluate  production  techniques,  materials,  cost-effectiveness  of 
varying  levels  of  artistic  moulage  features  were  conducted.  We  reached  a  stable  selection  of 
silicones  for  casting  the  skin  elements  (Smooth-On,  Inc.,  Macungie  PA,  EcoFlex  0030  tinted  with 
SilcPig  for  skin/eyelids).  Installation  of  realistic  eyelashes  is  desirable  to  the  surgeons  (to 
replicate  the  tangling  that  often  occurs  with  suture  threads),  however  it  is  a  highly  labor 
intensive  process. 

•  Additional  anatomical  features  were  added,  notably  a  tarsal  plate  layer  in  response  to  the  SMEs' 
criticism  of  the  original  single-layer  eyelid  (Smooth-On,  Dragon  Skin  10).  The  tarsal  plate 
underwent  a  series  of  revisions,  to  include  reinforcing  mesh  and  multiple  tests  of  different 
durometer  silicones  to  better  approximate  the  correct  feel  of  a  needle  penetrating  the  tissue 

•  The  head-form  skin  layer  was  modified  to  correct  a  gap  between  facial  skin  and  the  module  skin 
in  the  original  system. 


Figure  4:  Top  row:  experiments  with  use  of  alternate  silicone  materials  to  produce  correct  palpation  response  of 
skin,  explore  cost-benefit  relationship  of  increased  realism  (e.g.  eye  lashes,  brows);  original  gap  in  face/module 
margin.  Bottom  row:  modified  molds  showing  pre-molded  tarsal  plate  installed  for  overmolding  process;  back  side 
of  lid  laceration  module  showing  higher  stiffness  tarsal  plate  structure;  front  side  of  lid  laceration  module  after 
suturing  the  full  thickness  "wound". 

During  Q4,  initial  efforts  focused  on  preparations  for  conducting  user  data  collection  at  the  USUHS  eye 
trauma  course: 

•  A  standardized  lid  laceration  feature  was  built  into  the  molds  to  ensure  repeatability  in  creation 
of  the  trauma  modules  (allowing  for  more  direct  numerical  comparison  between  trainees). 

•  Small  "dimples"  were  added  to  the  skin  surface  to  provide  calibration  points  for  the  tracking 
system  to  allow  registration  between  the  physical  module  and  the  mathematical  model  used  to 
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define  Regions  of  Interest  (see  section2.3.2.b.v)  relevant  for  different  phases  of  a  surgical 
procedure. 

•  Positive  attachment  points  were  added  to  improve  repeatability  in  module  installation  and 
eliminate  the  possibility  of  module  shifting  during  the  course  of  a  scenario. 

•  Molds  were  revised  to  streamline  the  casting  and  module  production  process 

•  A  set  of  25  modules  was  produced  for  use  in  the  data  collection  trials  at  the  annual  Eye  Trauma 
course  at  USUHS  in  May  2012. 


Figure  5:  (left  to  right)  Y1  module  design  with  dimples  for  position  calibration  -  instruments  contact  at  each 
location  to  confirm  position  and  orientation;  attachment/alignment  features  to  maintain  stability  of  module  during 
scenario;  installation  of  module;  25  modules  prepared  for  USUHS  user  testing. 


2.3.2.b.iii.  Sensorized  surgical  instruments 

The  original  proof-of-principle  system  made  use  of  four  available  stainless  steel  instruments  lent  to  the 
team  by  Dr.  Mazzoli,  and  a  pair  of  Ascension  model  800  sensors  attached  to  the  user's  fingers  using 
Velcro  straps.  It  was  clear  from  discussions  with  the  ocular  surgeons  that  a  larger  suite  of  instruments 
was  necessary,  with  different  instruments  available  for  oculoplastics  (lid  repair)  and  for  the 
corneoscleral  procedures  (typically  finer  tipped,  smaller  instruments).  Further,  attachment  of  the 
position  sensors  to  the  user's  index  fingers  was  of  limited  usefulness  in  measuring  the  gestures 
performed  by  the  surgeon  -  estimating  instrument  tip  position  and  orientation  would  be  essentially 
impossible,  since  (in  our  observations  of  live  operations)  surgeons  often  reposition,  rotate  and  even 
invert  the  instruments  in  their  hands. 

Over  Yl,  we  broadened  the  set  of  instruments  available  from  4  to  9  plus  a  surgical  sponge  "spear".  We 
created  rigid  attachments  for  the  position  tracking  sensors  (now  smaller)  to  the  instruments,  but  added 
additional  sensing  capabilities,  signal  conditioning,  hot-swappability,  and  automated  instrument 
identification  by  the  data  acquisition  system. 

We  continued  to  make  use  of  the  Ascension  trakSTAR  magnetic  tracking  system,  however  we  did 
consider  alternate  sensing  technologies.  The  primary  alternative  is  optical  tracking.  These  systems,  to 
achieve  measurement  of  instrument  position  and  rotation,  typically  require  the  attachment  of  an  array 
of  optical  markers  or  retro-reflectors,  arranged  in  a  known,  3D  arrangement,  and  ideally  the  mounting 
of  multiple  tracking  cameras  around  the  system.  For  our  purposes,  and  given  the  observed  instrument 


18 


manipulations,  we  concluded  that  the  tracking  markers  would  interfere  with  the  surgeon's  normal 
manipulations  more  than  the  attachment  of  a  flexible  cable,  and  that  mounting  multiple  cameras  would 
be  more  complex  for  ultimate  end  users  than  a  single  magnetic  field  emitter.  The  clear  advantage  of 
optical  systems  over  magnetic  is  immunity  to  sensing  distortion  from  the  presence  of  nearby  ferrous 
materials,  however  we  designed  the  system  with  as  few  ferrous  elements  as  possible. 

Described  in  later  sections,  we  also  reviewed  the  main  competitor,  the  Polhemus  (Colchester,  VT) 
FASTTRAK  system,  to  evaluate  alternative  sensor  cable  properties.  During  most  of  this  program, 
Polhemus  did  not  have  a  micro  sensor  available  to  match  the  small  sizes  of  Ascension's  system,  however 
their  cables  were  substantially  more  pliable.  Only  at  the  end  of  the  project  did  Polhemus  release  an 
equivalent  sensor,  however  as  described  below,  the  Ascension  price  structure  continued  to  be  more 
advantageous  and  the  difference  in  cable  compliance  was  determined  by  users  to  be  not  of  sufficient 
significance  to  warrant  changing  technologies. 

The  following  figures  show: 

•  Expansion  of  the  instrument  suite  from  four  to  nine  (with  additional  instruments  added  in  later 
years)  and  acquisition  of  titanium  instruments  (vs.  steel)  to  minimize  field  distortion  due  to  the 
instruments  (see  Figure  6) 

•  CAD  modeling  of  the  instruments  to  provide  graphical  models  for  integration  into  the  data 
acquisition/user  interface  software  and  support  development  of  connectors/sensor  holders  and 
finite  element  analysis  of  the  instruments  (Figure  7) 

•  Attachment  of  strain  gages  at  critical  locations  to  measure  the  flexing  of  the  instruments  as  the 
jaws  are  closed,  and  measure  the  deflection  of  forceps  tips  when  grasping  tissue  or  upon 
closure,  and  a  prototype  signal  conditioning  system  and  interface  to  the  data  acquisition 
software  with  the  current,  miniaturized  equivalent  version.  The  finest  tipped  forceps  have  been 
equipped  with  a  silicon  strain  gage  sensor  (not  shown),  much  smaller  than  conventional  foil 
gages,  so  that  force  sensing  can  be  made  even  on  the  very  gentle  manipulations  during  corneo¬ 
scleral  manipulations  (Figure  8) 

•  A  series  of  tracking  sensor  mounts,  from  an  early  fixed  system  for  testing  (which  would  require 
nine  separate  tracking  sensors  -  Figure  9),  an  initial  version  allowing  interchangeability  of  the 
tracking  sensors  between  instruments  (allowing  for  just  three  position  tracking  sensors  to  be 
necessary  -  Figure  11),  a  later  version  integrating  an  onboard  signal  conditioning  circuit  sending 
a  pre-amplified  signal  back  to  the  data  acquisition  system,  a  revision  of  the  miniature  circuit 
board  to  minimize  self-heating  and  single  drift  (Figure  10),  and  a  version  which  includes  an  array 
of  mounting  points  so  that  the  user  can  interchange  the  connector  using  only  one  hand  (Figure 
11  -  later  versions  are  described  in  2.3.4.c). 

•  An  initial  prototype  data  acquisition  board  using  an  Arduino  kit  microcontroller  system,  followed 
by  an  improved,  USB-compatible  custom  board  which  is  expandable  to  allow  multiple  functions 
to  be  sensed  or  controlled  over  the  single  USB  connection  (Figure  10). 

•  Elastic  wrist  straps  with  cable-management  clips  attached.  We  developed  these  for  the  users  to 
wear  to  help  minimize  sensor  cable  tangling  during  a  scenario.  One  sensor  cable  would  run 
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through  the  clip  on  each  arm,  with  the  third  sensor  cable  available  for  instruments  left  on  the 
array  tray  (Figure  9  and  Figure  11).  This  cable  configuration  was  retained  throughout 


development  of  the  system. 


Figure  6:  (left)  Original  steel  instrument  suite,  (right)  New  titanium  instrument  suite  (to  scale) 


Figure  7:  CAD  models  of  instruments  for  use  in  visualization  tools,  finite  element  analysis 
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Figure  8:  (top  left,  center)  Finite  element  analysis  (FEA)  of  forceps  and  scissors  showing  locations  of  high  strain, 
optimal  for  strain  gauge  placement  for  measuring  grasping/closure  action,  (top  right)  FEA  showing  location  of  high 
strain  under  tip  loading  for  measuring  applied  force,  (bottom)  Strain  gauges  attached  to  three  forceps;  breadboard 
test  circuit  for  amplifying  strain  gauge  output  signals. 
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Figure  9:  (Top)  Original  finger-mounted  position  tracking  sensors;  version  1  of  sensor  mounting  rigidly  attached  to 
instruments  directly  for  improved  tracking  accuracy.  (Bottom)  Version  2  of  sensor  mounting  fixtures  with 
swappable  connectors  and  wrist  straps  for  cable  management;  detail  of  version  2  connector  to  needle  holder. 


Figure  10:  Miniature  circuit  boards  for  strain  gauge  amplification  and  instrument  identification  (larger  than  actual 
size);  Data  acquisition  board  for  strain  gauge  signals,  board  identification. 
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Figure  11:  (top)  Version  3  connectors  with  mini  circuit  boards  in  use  by  Dr.  Mazzoli  at  MMVR  2012.  (middle)  Suite 
of  instruments  with  swappable,  auto-ID  instruments;  detail  of  connector  sensors  connect  to  all  instruments 
regardless  of  instrument  configuration  (i.e.  forceps  vs.  scissors  vs.  needle  holder);  wrist  strap  provides  some 
improvement  to  prevent  cable  tangling,  (bottom)  Simulator  with  rack  of  instruments  ca.  2012. 

Following  the  2012  USUHS  testing,  we  identified  areas  of  connector  weakness  and  instability  of  the 
sensor  with  respect  to  some  of  the  instruments.  In  addition,  this  iteration  of  the  one-handed 
instrument  holder  tray  was  found  to  allow  the  user  to  poke  themselves  on  the  sharp,  downward 
pointing  instruments.  Further,  the  prismatic  features  that  mate  between  the  instrument  connectors 
and  the  holder  tray  can  jam  if  the  user  angles  an  instrument  while  picking  it  out  of  the  tray  -  a 
somewhat  non-intuitive  grasp  and  removal  technique  is  necessary  to  avoid  jamming,  with  the  potential 
for  breaking  the  connector  shell.  Further  evolution  to  address  these  issues  is  described  in  later  sections. 

2.3.2.b.iv.  Simulator  stand,  camera/kinect/projector/web  cam  mounting 
The  original  proof  of  principle  system  integrated  the  magnetic  field  emitter  (Ascension  Technology, 
short  range  emitter)  into  the  throat  area  of  the  head  form.  This  was  not  optimal  as  it  placed  the  most 
sensitive  area  of  the  magnetic  field  above  the  mannequin's  forehead  rather  than  near  the  eyes.  In 
addition,  our  improved  system  supplements  the  position  tracker  with  video  recording,  possible  optical 
hand  motion  sensing  using  a  Kinect  sensor  (available,  and  partially  integrated  into  the  full  gesture 
detection  system),  a  video  projector  for  feedback  and  augmented  reality  applications.  For  this  reason, 


23 


we  developed  and  incrementally  improved  a  framing  system  that  holds  the  head  form  in  place  relative 
to  the  repositioned  field  emitter. 

Early  testing  made  use  of  convenient  office  furniture  as  mounting  hardware  for  a  miniature  video 
projector  and  a  miniDV  video  recorder,  followed  by  development  of  the  first  version  of  the  a  mounting 
stand,  initially  with  adjustable  height  projector/Kinect  mounting  and  instrument  tray,  and  later  with  an 
adjustable  brightness  lighting  system,  the  improved  instrument  holding  array  described  above,  and  a 
mounting  for  a  web  cam  video  input  (for  simplified  synchronization  of  video  recording  and  sensor 
inputs,  vs.  later  synchronization  of  mini-DV  tape  recordings  that  would  need  to  be  reimported  for 
analysis). 


Figure  12:  (top)  Early  test  configuration;  initial  stand  design  with  rigid  mounting  of  head  form;  Kinect  and  projector 
mounting  above  head  form,  (bottom)  First  assembly  of  stand;  configuration  of  system  used  at  USUHS  user  trials. 

At  the  end  of  Yl,  the  stand  had  reached  an  initial  design  freeze  for  the  USUHS  trials.  In  response  to 
observations  and  feedback  at  the  trials,  modifications  were  made  and  additional  features  continued  to 
be  added  throughout  the  program,  as  described  in  section  2. 3.4. a. 

2.3.2. b.v.  Data  collection  user  interface  development,  calibration,  linearization, 
development  of  Event-Driven  Gesture  Segmentation  and  Task  Recognition 
system  (EDGSTR) 

The  initial  suite  of  software  tools  and  architecture  that  enable  data  acquisition,  recording,  playback,  and 
analysis,  and  that  controls  the  projection  and  lighting  systems  were  rewritten  from  the  ground  up, 
compared  with  the  original  proof-of-principle  system.  As  the  system  hardware  evolved,  from  a  small 
number  of  fixed  position  tracking  sensors,  to  a  large  number  of  sensors  (one  per  instrument),  to  three 
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interchangeable  sensors,  to  the  inclusion  of  the  strain  gage  sensors,  to  an  increasingly  capable  system 
for  defining  surgical  gestures  and  events  and  providing  meaningful  displays  of  data,  the  software 
continued  to  evolve  with  it.  The  series  of  images  below  shows  various  phases  of  the  revisions  of  the 
software  system: 

•  Basic  reimplementation  of  6  degree  of  freedom  tracking  system  data  acquisition  and 
synchronization  of  virtual  representation  with  physical  head  form  and  instruments  (Figure  13) 

•  Definition  of  Regions  of  Interest  (ROIs),  spherical  zones  centered  at  surgically  relevant  sites  (to 
be  defined  by  surgical  trainers/experts)  to  allow  detection  of  events  such  as  entry  or  exit  from 
these  zones  (Figure  14) 

•  Graphical  display  of  strain  gage  sensor  inputs,  representation  of  three  tools  tracked 
simultaneously  (Figure  14) 

•  Refinement  of  data  stream  presentation  (orientation,  grasping,  force)  for  all  nine  instruments 
simultaneously,  creation  of  dynamic  playback  system  for  use  in  post-processing  and 
performance  analysis  (Figure  15) 

•  Example  of  identifiable  events  based  on  raw  strain  gage  information,  showing  applied  force 
during  needle  insertion  (red,  upper  band),  and  cutting  of  suture  (short  red  blocks,  lower  band) 
(Figure  15) 

•  Revision  to  track  swappable  instruments,  using  the  automatic  instrument  identification  function 
of  the  miniature  circuit  boards  attached  to  each  instrument  -  fewer  distinct  bands  of  data  are 
presented  on  the  streaming  display  as  the  identity  corresponding  with  the  input  data  switches 
depending  on  the  instrument  in  use.  All  data,  including  instrument  identity,  are  recorded  to  file 
for  later  analysis  at  this  phase  (Figure  16) 

•  Integration  of  the  Kinect  data  stream  into  the  user  interface  (blue/gray  window  at  left  of  screen 
shot)  (Figure  16) 

•  Results  plot  showing  example  of  global  performance  variable  levels  (time,  path  length,  motion 
smoothness,  orientation  datum)  (Figure  17) 

•  Wireless  numeric  keypad  used  for  manual  event  recording  for  system  training.  To  develop 
event/gesture  detection  algorithms,  we  provided  a  means  to  manually  identify  events/gestures, 
for  comparison  with  sequences  of  automatically  detected  low-level  events  and  determination  of 
relevant  numerical  thresholds.  (Figure  17) 

•  Screen  shot  showing  low  level  event  detection  window  (text  window  at  left),  complex  set  of 
Regions  of  Interest  overlaid  on  head-form  representation.  The  EDGSTR  system  is  described  in 
more  detail  below,  and  in  the  IMIASH  workshop  paper  [De  Novi  et  al.,  2012]  in  section  7.4  of  the 
Appendices  (Figure  18). 
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Figure  13:  instrument-mounted  sensors  and  correspondence  between  real  and  virtual 


Figure  14:  Regions  of  interest  defined  near  eye  zones;  measurement  and  display  of  grasping/cutting  action  and 
force  measurement 


Figure  15:  Data  acquisition  interface  showing  parallel  tracks  of  orientation,  force,  grasping/cutting  actions;  Detail 
of  tracks  showing  forceps  (top  color  bar)  grasp  (red)  and  release  (blue),  scissors  cutting  (lower  color  bar,  red) 


Figure  16:  Reduced  display  of  tracks  for  3  swappable  instrument  measurements.  Interface  with  addition  of  Kinect 
depth  map  measurement/display 
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Figure  17:  Early  global  score  output  screen;  manual  input  keypad. 


Figure  18:  Recent  interface  screenshot  showing  regions  of  interest  around  eye,  around  instrument  tray,  with 
timestamped  listing  of  ROI  transitions  recorded  at  left. 


In  addition  to  data  collection  functions,  software  tools  allow  calibration  of  the  sensors  within  the 
magnetic  field  and  compensation  of  the  sensor  measurements  against  distortions  caused  by  static 
environmental  influences  (large,  nearby  volumes  of  metal).  A  sensor  mounting  frame  is  shown, 
together  with  a  representation  of  the  physical  grid  used  for  calibration  (green  "block")  and  the 
measured  apparent  positions  of  the  sensor.  By  mapping  the  measured  results  back  to  the  known 
geometric  positions,  an  improved  estimation  of  instrument  positions  can  be  obtained. 


Figure  19:  Magnetic  tracker  calibration  system;  graphical  display  of  nominal  positions  (green)  and  measured 
positions  (red)  to  be  mathematically  remapped  to  correct  locations. 
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Towards  the  end  of  Year  1,  following  the  Course,  we  began  integrating  these  elements  of  the  software 
to  make  best  use  of  the  video  and  instrument  motion  data  collected  at  the  2012  USUHS  Ocular  Trauma 
Course  and  in  future  tests. 

A  software  library  was  under  development  to  allow  recording  of  video  sequences  together  with  data 
collection,  using  a  web-cam  as  a  source,  or  to  import  separately  recorded  video  from  a  miniDV 
camcorder.  Video  is  stored  in  compressed  AVI  format  and  can  be  displayed  with  playback  of  the  raw 
data  to  provide  a  complete  picture  of  the  recorded  procedure.  After  the  2012  Course,  we  completed 
synchronized  display  of  the  video  stream  with  the  rest  of  the  data  acquired  during  the  procedure  in 
order  to  provide  a  consistent  playback,  despite  differences  in  sampling  rates  of  the  instrument  paths, 
instrument  grasping/force  sensing  and  the  frame  rate  of  the  video. 

During  this  period,  we  developed  the  initial  versions  of  our  analysis  tools,  presenting  the  streamed  data 
in  the  form  of  the  multi-track  time  line.  Shown  below,  the  first  track  shows  the  event  labels  for  the 
manually  input  keystrokes  that  were  recorded  during  the  trials  corresponding  with  a  set  of  specific 
events  (cuts,  grasps,  throws,  etc.)  Displaying  keystrokes  in  the  timeline  makes  it  is  easy  to  find  specific 
segments  of  a  given  scenario  to  extract  useful  information  for  analysis  as  we  identified  the  geometric 
and  activity  thresholds  for  later  automatic  detection.  Data  manipulation  capabilities  were  added  to  the 
timeline,  allowing  the  user  to  cut,  clean,  segment,  and  export  sections  of  data.  The  data  manipulation 
feature  was  important  to  create  to  extract  and  analyze  specific  sections  with  appropriate  statistical 
tools,  leading  to  creation  of  libraries  of  patterns  that  would  later  be  used  to  generate  the  gesture 
recognition  engine.  All  the  segmented  data  can  be  exported  in  several  file  formats  that  can  be  loaded  by 
different  commercial  applications  (e.g.  Microsoft  Excel). 

With  segmented  data  identified,  we  implemented  functions  to  measure  parameters  including  total  task 
time,  total  instrument  path  length,  and  compactness  of  motion,  both  in  global  terms  and  also  by 
instrument. 

We  also  refined  the  position  tracking  system  calibration  process,  developing  code  that  makes  use  of 
instrument  motions  centered  at  the  calibration  "dimples"  on  the  lid  laceration  portal  (section  2.3.2. b.ii). 
This  ensured  accuracy  and  precision  of  tracking  for  the  scenario  in  the  vicinity  of  the  eye,  eliminating  any 
shifting  between  the  ideal  location  of  the  magnetic  field  emitter  and  the  physical  head  due,  for  example, 
to  transportation  or  bumping  the  structure. 
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Figure  20:  Detail  of  instrument  closure  traces  with  new  indication  (top  row  "Keystrokes")  of  manually  entered 
events  to  facilitate  comparison  of  surgical  gestures  with  measured  data. 

2.3.2.b.vi.  Content  presentation  architecture  and  content  development 
During  Yl,  we  began  to  develop  a  learning  architecture  to  integrate  the  simulator  with  curricular 
elements.  We  conducted  a  review  of  the  prior  codebase  used  for  the  user  interface  of  the  CELTS 
system,  a  Computer  Enhanced  Laparoscopy  Training  System  that  we  developed  in  the  mid-2000s 
[Stylopoulos  et  al,  2004].  The  CELTS  interface  was  developed  in  Java  Server  Pages  (JSP).  JSP  technology 
has  now  been  superseded  by  more  common  scripting  languages  such  as  PHP  and  the  advent  of  the 
newer  HTML5.  Therefore,  we  redesigned  the  interface  architecture  around  these  newer  languages 
while  retaining  some  of  the  structure  of  the  content  used  in  the  earlier  system. 

The  overarching  goal  of  this  effort  was  to  coherently  link  our  simulation  development  efforts  with 
military  and  civilian  learning  objectives.  We  aimed  to  provide  an  architecture  that  supports  familiarizing 
practitioners  with  the  correct  understanding  of  the  tools,  steps,  tasks  and  procedures  specific  to  ocular 
and  facial  trauma,  both  simple  and  complex. 

Figure  21  shows  the  early  evolution  of  our  concepts  for  progression  through  a  scenario,  storage  of 
performance  data,  tools  for  scenario  and  scenario  library  development.  Figure  22  shows  an  example  of 
an  instructional  screen  for  display  during  the  simulator  set-up  process. 
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Figure  21:  (top)  early  concept  sketches  for  scenario  generation,  storage,  presentation,  branching  sequences  of 
measurement  and  feedback,  recording  of  performance  data,  analysis  and  scoring  of  performance,  (bottom) 
refinement  of  concepts  guiding  Year  2  development. 
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Figure  22:  Typical  instructional  screen  in  the  Year  1  era,  with  progress  through  scenario  highlighted  at  left. 


2.3.3.  “Month  4”  - 


2. 3. 3. a.  Perform  user  evaluation  at  USUHS  TriService  Ophthalmology  courses. 

One  of  the  most  important  accomplishments  of  year  1  was  conducting  our  first  major  user  testing 
exercise.  The  original  calendar  for  this  research  program  envisioned  a  start  date  in  January  of  2011, 
testing  of  the  revised  prototype  would  occur  early  in  the  program. 
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Contact  established  at  the  2011  AFSIM  meeting  with  Mr.  Shad  Deering  lead  ultimately  to  contact  with 
Dr.  (COL)  Richard  Stutzman,  the  then  ophthalmology  program  director  at  WRNNMC.  Early  discussions 
with  Dr.  Stutzman  indicated  that  user  testing  might  be  conducted  during  regularly  scheduled  academic 
days  at  WRNNMC  and/or  Ft.  Belvoir,  when  staff,  fellow  and  resident  participation  would  not  interfere 
with  patient  duties.  The  human  studies  protocol  that  drafted  by  Dr.  Paul  Neumann,  formerly  of  our 
group  for  testing  the  original  proof-of-concept  system,  was  adapted  to  reflect  these  plans  and  the 
approval  process  was  commenced.  Working  with  the  MGH  IRB,  Dr.  Jeffrey  Stephenson  at  TATRC  and  the 
ORP  HRPO,  we  made  changes  to  the  protocol,  consent  form,  procedure  for  recruiting  subjects  and 
information  presented  to  potential  participants.  As  this  process  continued,  Dr.  Stutzman  proposed  that 
ultimately  it  would  be  most  efficient  and  practical  to  conduct  our  testing  exclusively  at  USUHS  during 
the  annual  Ocular  Trauma  course  in  May,  which  allowed  sufficient  time  to  prepare  the  simulator  for 
such  use  and  to  obtain  all  of  the  necessary  approvals. 

The  MGH  IRB  determined  that  the  protocol  constitutes  minimal  risk  to  participants  and  processed  it 
using  expedited  review.  The  ORP  HRPO  conducted  a  similar  review.  USUHS  determined  that  since  the 
trauma  course  was  being  conducted  on  campus,  but  that  no  USUHS  personnel  would  be  directly 
involved  in  the  conduct  of  the  study  except  as  subjects,  the  USUHS  IRB  would  not  require  review.  A 
vendor  agreement  was  established  to  permit  the  transport  of  the  simulator  to  the  USUHS  campus  and 
describe  the  permissions  and  responsibilities  of  the  parties  involved. 

2.3.3. a.i.  Preliminary  user  testing  with  SMEs 

In  the  period  prior  to  the  formal  user  testing  at  USUHS,  we  conducted  a  series  of  demonstrations  to  our 
SMEs  (Drs  Freitag,  Kanoff  and  Lefebvre  of  MEEI,  Dr.  Mazzoli)  and  their  colleagues,  to  obtain  early 
feedback  for  system  improvement,  to  define  the  nature  of  the  exercises  to  be  performed  by  users,  and 
to  ensure  that  all  of  the  necessary  supplies  would  be  available  to  users  (e.g.  type  and  number  of  sutures, 
auxiliary  supplies  such  as  gloves).  During  these  demonstrations,  we  asked  the  experts  to  perform  the 
complete  process  of  full  thickness  upper  eye  lid  laceration  repair,  from  exploration  of  the  simulated 
injury  through  to  ensuring  neatness  of  cut  ends  of  sutures  at  the  end  of  the  procedure.  We  also  asked 
them  to  perform  a  series  of  15  elemental  exercises  so  that  we  could  collect  data  on  the  performance  of 
different  knot  styles,  suture  needle  handling  technique  and  other  gestures/tasks  we  expected  to 
observe  during  the  full  laceration  scenario.  These  tests  allowed  us  to  estimate  the  time  required  to 
complete  different  scenario  combinations,  which  lead  to  the  determination  that  at  the  USUHS  user 
testing,  we  would  ask  participants  to  perform  only  the  lid  laceration  repair,  without  additional  elemental 
exercises,  as  this  would  place  an  undue  demand  on  participants'  time  during  the  tightly  scheduled 
course  agenda. 

2.3.3. a.ii.  User  data  collection  at  2012  USUHS  Ocular  Trauma  courses 

The  first  set  of  user  trials  were  conducted  over  23-24  May  2012,  during  which  time  21  participants 
started  the  data  collection  exercises,  and  20  completed  the  full  consent  and  scenario  sequence.  These 
included  12  attending  surgeons  of  various  disciplines,  two  fellows  and  six  residents.  We  planned  to  have 
a  larger  representation  of  earlier-career  surgeons  in  the  study,  however  the  time  limitations  of  the 
Course  did  not  permit  them  significant  availability  away  from  their  small-group  lectures  and  wet-lab 
sessions.  The  senior  surgeons  and  instructors  had  significantly  more  availability. 
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During  the  sessions,  only  skill  level  and  handedness  were  recorded  as  demographic  data,  consistent  with 
the  anonymization  process  of  the  protocol.  In  later  years,  described  below,  we  expanded  upon  the  list 
of  participant  characteristics.  We  collected  6DOF  instrument  motion  data,  instrument  exchanges,  video 
recordings  of  hand  motions  in  the  area  of  the  mannequin's  face  (Figure  23),  and  then  manually  entered 
surgical  event  keypad  entries.  We  separately  made  notes  when  users  made  comments  or  suggestions 
on  how  to  improve  or  alter  the  system,  without  attaching  name  or  rank  information  to  those  comments. 
Observations  of  the  ease  or  difficulty  of  use  of  the  system,  system  failure  modes  and  system  damage 
were  made  so  that  we  would  be  able  to  improve  future  iterations  of  the  system. 


Figure  23:  Video  frames  from  USUHS  trial  participant. 


2.3.3.a.iii.  Other  user  trial  outcomes 

At  the  trial,  Dr.  Mazzoli  observed  that  we  were  collecting  a  series  of  used  trauma  modules,  which 
showed  suture  patterns  for  experts  and  novices,  all  on  identical  injuries.  As  multiple  physicians  would 
never  all  be  able  to  perform  surgery  on  identical  injuries,  even  in  cadaver  or  animal  lab  situations,  Dr. 
Mazzoli  suggested  performing  an  analysis  on  our  trauma  modules  to  determine  if  there  are  measureable 
differences  in  suture  patterns.  We  created  a  lighting/mounting  jig  to  allow  us  to  generate  aligned 
photographs  of  all  of  the  used  modules,  however  the  ongoing  development  requirements  and  shifts  in 
analysis  methods  took  precedence  over  this  analysis. 


2.3.3. b.  Locate  and  coordinate  commercial  partner  to  manufacture  first  version  of  the 
stand-alone  eye  trauma  simulator. 

This  section  describes  most  of  the  efforts  towards  commercialization  that  were  made  throughout  the 
program,  not  only  those  in  the  early  months  of  the  program. 


Initial  commercialization  discussions  comprised  contact  with  simulator  vendors,  including  Laerdal 
(Wappingers  Falls,  NY)  and  CAE-Healthcare  (Sarasota,  FL),  in  the  context  of  acquiring  one  of  their 
mannequin  systems  for  the  Phase  3  element  of  this  work,  with  the  intent  of  developing  a  trauma  head 
extension  for  the  systems  of  the  mannequin  manufacturers.  In  the  case  of  Laerdal,  they  conveyed  that 
they  have  worked  with  developers  creating  systems  compatible  with  their  own,  working  closely  when 
the  systems  align  with  their  own  interests.  We  continued  to  maintain  contact  with  Laerdal  in  the 
context  of  integration  of  our  software  and  their  mannequin  application  programmer's  interface  (API), 
although  their  interest  in  the  system  did  not  increase.  Resulting  from  the  acquisition  of  METI  by  CAE- 
Healthcare  early  during  this  program,  discussions  with  CAE  were  with  the  METI  team  rather  than  the 
earlier  Caesar  development  team  we  came  to  know  during  the  MGH  licensing  and  tech  transfer  of  our 
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COMETS  system  [Bardsley  et  al,  2014].  Early  conversations  showed  that  there  is  no  equivalent  software 
tool  to  Laerdal's  API  for  the  METI  mannequins,  however  should  they  be  interested  at  a  later  time 
integrating  our  head  form  and  software  with  theirs,  our  final  system  has  well  documented  interface 
protocols  and  physical  connectivity  that  simplifies  any  modifications  that  they  would  find  necessary. 

We  evaluated  the  use  of  Phillips  Studio  (Redland,  Bristol,  UK)  products  for  our  system  and  also  discussed 
their  manufacturing  capabilities,  however  Phillips  was  not  in  a  position  to  pursue  a  simulator  with 
advanced  computing  and  sensing  capabilities. 

As  described  in  2.3.5. b,  with  respect  to  the  augmented  reality  microscope  system,  Scan  Optics  Pty  Ltd 
(Adelaide,  Australia)  expressed  interest  in  helping  us  to  develop  the  optics  for  the  integrated  display, 
however  their  market  focus  appeared  to  be  production  of  microscopes  for  the  surgical  end  user,  rather 
than  entering  the  training/simulation  space.  As  described  below,  we  made  significant  progress  in 
developing  augmented  display  optics  for  our  prototype. 

An  invention  disclosure  was  filed  on  the  original  (pre-phase  1)  proof-of-concept  system,  although  MGH 
had  not  pursued  a  patent  application  as  Year  1  came  to  a  close.  The  original  system  had  been  publicly 
disclosed  prior  to  the  commencement  of  this  work,  and  early  improvements  to  the  stand-alone  system 
were  considered  to  be  relatively  obvious  to  one  skilled  in  the  art,  in  light  of  simulators  for  other 
procedures  that  also  use  tracking  technology  and  anatomical  models.  Value  to  a  vendor  at  this  stage 
would  have  been  primarily  in  the  know-how  generated  in  the  course  of  system  development.  MGH  has 
previously  licensed  technology  in  which  know-how  is  a  substantial  portion  of  the  value,  so  we  did  not 
see  this  as  an  obstacle  at  this  time,  but  did  continue  to  record  and  report  the  development  of  novel 
technological  elements  of  the  system. 

Subsequently,  in  support  of  a  strong  licensing  agreement  when  a  commercial  partner  is  identified,  a 
provisional  patent  application  was  submitted  to  the  USPTO  prior  to  the  2013  MHSRS  meeting  to  allow  us 
to  present  new  features  and  inventive  content.  At  the  end  of  Year  3,  we  were  working  with  patent 
attorneys  to  make  the  conversion  to  full  patent  application. 

At  the  2012  USUHS  user  trials,  some  of  the  eye  trauma  specialists  indicated  that  they  would  find  the  eye 
lid  modules  useful  in  their  training  responsibilities,  so  we  began  to  contemplate  alternative 
commercialization  options,  including  sales  of  the  trauma  modules  as  consumables  for  the  instrument 
tracking  and  simulation  system  or  as  individual  components. 

After  the  2012  USUHS  user  trials,  we  analyzed  the  current  commercial  trainer  product  lines  available  for 
ocular  surgery  training.  This  analysis  served  to  help  evaluate  our  options  for  transitioning  the  current 
research  prototype  to  a  marketable  product.  Throughout  the  course  of  the  project,  we  informally  asked 
the  end  users  which  training  systems  they  currently  use.  The  most  advanced  simulator  that  trainers 
referred  to  was  the  Eyesi  Surgery  Simulator  (VRmagic  Holding  AG,  Germany).  At  approximately  $200k 
per  unit,  this  simulator  is  substantially  more  expensive  than  the  next  type  of  simulator  used,  namely,  the 
silicone  eyes  models  manufactured  by  Phillips  Studio.  The  disposable  eye  models  from  Phillips  can  be 
used  in  the  Eyesi  unit  and  cost  approximately  $75  each.  It  can  be  clearly  seen  that  there  is  a  substantial 
gap  in  the  current  product  offering.  However,  due  to  the  comparatively  small  number  of  ocular 
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surgeons,  the  number  of  commercial  players  in  this  market  is  correspondingly  small.  This  does  not 
diminish  the  opportunity  to  deliver  a  mid-range  hybrid  simulator  that  combines  realistic  disposable 
anatomical  modules  with  tracked  instruments  and  dynamic  guidance. 

Following  inquiry  with  FCI  Ophthalmics  (Pembroke,  MA),  an  instrument  and  device  company  specializing 
in  ophthalmology,  in  regards  to  ordering  canalicular  stents,  we  established  an  NDA  with  Ms.  Anne  Bozak 
and  FCI  in  Q9.  This  allowed  us  to  have  discussions  with  Ms.  Bozak  and  her  colleague  Michele  Joyce  in 
Qll.  Their  position  at  the  time  was  that  there  were  no  good  commercial  simulators  that  could  support 
canalicular  repair  training,  and  they  expressed  interest  in  evaluating  our  system  as  it  evolved.  A  follow¬ 
up  meeting  with  Ms.  Bozak  and  Mr.  Jim  Perez,  their  director  of  sales  for  North  America  was  held  at  the 
end  of  Year  3.  Subsequently,  as  described  below,  they  were  present  for  a  review  of  the  anatomical 
modules  in  the  final  months  of  the  program  and  remain  interested  and  await  a  number  of  minor 
modifications  that  we  intend  to  make  following  completion  of  this  project. 

A  confidentiality  agreement  process  was  established  with  NVIS,  Inc.  (Reston,  VA),  a  company  that 
develops  VR  and  augmented  reality  head  mounted  displays  and  binoculars,  in  July  2014.  They  saw  the 
abstract  for  our  MMVR  2014  poster  but  were  unable  to  meet  us  at  the  conference.  A  conference  call 
was  held  in  November  2014  with  Minoo  Bablani,  CTO  and  VP  Product  Development,  to  discuss  the 
technology  that  we  developed.  They  have  interest  in  developing  medically-related  products,  having 
previously  bid  on  the  development  of  a  VR-based  cataract  surgery  simulator.  Our  prototype  system, 
which  by  2014  combined  graphical  and  directly  observed  real  imagery,  sparked  interest  for  NVIS  in 
regards  to  a  variety  of  microsurgery  applications.  Contact  with  NVIS  will  be  made  again  following  the 
final  changes  to  the  microscope  system  and  demonstration  interface. 

An  NDA  established  with  Lantos  Technologies,  Inc.  (Wakefield,  MA)  in  Year  2,  towards  applying  the 
technology  and  methods  we  developed  for  fabricating  the  realistic  head  form,  lead  to  a  sponsored 
research  agreement  to  fabricate  a  new  simulator  for  training  in  the  use  of  an  ear  canal  3D  scanning 
instrument.  That  relationship  was  ongoing  at  the  end  of  the  program.  One  useful  benefit  of  the  work 
with  Lantos  was  the  opportunity  to  make  use  of  small  touch-screen  monitors  as  part  of  the  interface  for 
a  simulator.  Following  application  of  a  touch-screen  to  the  Lantos  system,  we  adapted  the  software 
used  for  that  element  of  the  Lantos  simulator  and  the  same  touch  screen  monitor  for  use  in  this  system. 

On  13  August  2014,  we  met  with  Dr.  David  Hunter,  Ophthalmologist-in-Chief  of  Boston  Children's 
Hospital  and  his  colleague  Dr.  Gena  Heidary,  who  is  responsible  for  their  strabismus  course.  They 
expressed  interest  in  viewing  the  completed  system  and  discussing  the  development  of  a  strabismus 
surgery  training  module.  This  is  likely  to  be  compatible  with  the  enucleation  module  that  will  be 
developed  under  CIMIT  funding  that  was  awarded  in  part  as  a  result  of  the  successes  of  the  current 
program. 

We  met  with  John  James,  Director  of  Engineering  at  Sawbones  (Pacific  Research  Labs),  on  15  December 
2014  to  discuss  possibilities  for  fabrication  and  commercialization  of  this  and  other  simulators. 
Sawbones  has  developed  some  additional  electronics  fabrication  capabilities  since  we  worked  with  them 
on  our  Virgil  Chest  Trauma  simulator  [Ritter  &  Bowyer,  2005]  however  the  Ocular-Facial  Trauma  system 
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remains  outside  their  area  of  specialty.  They  have  had  some  limited  experience  developing  an  eye 
surgery  system,  which  included  an  eye  globe  which  could  pivot  on  its  base,  suggesting  that  they  could  be 
a  potential  supplier  of  the  anatomical  components  of  the  system  for  another  system  integrator. 

At  the  2015  USUHS  Ocular  Trauma  Course,  following  the  formal  end  of  this  program,  Dr.  Marcus  Colyer, 
the  course  director,  introduced  our  team  to  that  of  Simulation  Systems,  Inc.  (Hummelstown,  PA)  (Dr. 
Joseph  Sassani  and  Mr.  Brian  Smith),  leading  to  extended  discussions  regarding  how  the  software 
libraries  that  the  MGH  Simulation  Group  staff  have  developed  may  be  applied  to  solve  certain  remaining 
issues  of  Dr.  Sassani's  virtual  reality-based  system.  In  addition,  we  began  to  explore  the  possibility  of 
taking  advantage  of  the  commercial  status  of  Simulation  Systems  (vs.  the  non-profit  academic  hospital 
status  of  the  Simulation  Group)  as  a  route  towards  commercialization  of  the  Simulation  Group 
simulator.  A  confidentiality  agreement  was  completed  in  June,  2015,  followed  by  Dr.  Ottensmeyer 
being  named  as  a  scientific  advisor  for  SimSys,  and  ongoing  efforts  towards  preparation  of  an  STTR  grant 
proposal  to  verify  and  validate  the  system. 

2.3.4.  “Month  6”  -  Revise  design  and  fabricate  improvements  based  upon  USUHS  Spring 
evaluation. 

This  section  describes  not  only  the  revisions  made  after  the  first  round  of  user  testing  at  the  2012 
USUHS  Ocular  Trauma  Course,  but  also  those  made  throughout  the  program  based  on  user  feedback 
and  observations  of  the  system  in  use. 

In  Year  1  preliminary  expert  testing  and  the  USUHS  trials  were  very  instructive  in  terms  of  identifying 
elements  for  system  improvement,  calling  for  system  modification  and  revision  to  the  testing  process 
for  future  user  trials. 

After  Year  1,  our  task  list  for  redesign  included  the  following,  most  elements  of  which  were  addressed  by 
the  completion  of  the  final  prototype  iteration: 

•  Strengthen  instrument  connectors  to  prevent  damage  during  extended  use,  modify  connectors 
to  make  connector/instrument  orientation  more  obvious  and  mating  easier 

•  Provide  adjustable  height  seating  and  alternate  loupe  magnification/working  distance  for  users 
of  different  heights 

•  Broaden  web-camera  field  of  view  to  include  instrument  tray  to  support  identification  of 
surgical  actions  away  from  vicinity  of  surgical  site 

•  Add  anatomical  features  to  modules,  including  eyelashes  if  possible,  softer  tarsal  plate 

•  Eyelid  tissue  should  include  redundant  tissue,  which  typically  buckles  and  rolls  under  when  the 
eyes  open,  rather  than  the  taught,  more  fixed  version  of  our  current  modules.  Versions  of  this 
were  developed  to  test  blinking,  but  not  implemented  in  the  modules  provided  for  the  USUSH 
courses,  as  not  essential  for  the  selected  laceration  training  scenarios. 

•  The  division  between  tarsal  plate  and  skin  should  include  a  pigmented  layer  representative  of 
the  orbicularis  orbis  muscle,  used  as  a  landmark  for  suturing 

•  The  tool  tray  should  be  reconfigured  so  that  sharp  ends  of  instruments  point  away  from  the 
user,  that  the  alignment/retention  features  of  the  tray  should  allow  for  "unwinding"  of  the 
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sensor  cable,  which  can  become  twisted  (as  with  a  telephone  handset  cord),  that  the 
alignment/retention  features  should  allow  for  simpler,  quicker  engagement/disengagement, 
with  reduced  possibilities  for  jamming  and  potentially  breaking  the  jammed  instrument 
connector 

•  A  means  for  identifying  when  sutures  penetrate  through  the  palpebral  conjunctiva  (inner  lining 
of  eyelid),  which  would  scratch  and  irritate  the  patient's  eye  surface  (not  implemented). 

•  The  system  stand  should  be  reconfigured  to  improve  the  location  of  the  magnetic  field  emitter, 
instrument  tray  location. 

•  Additional  consideration  for  the  participation  of  two  trainees,  one  serving  as  surgeon,  the  other 
as  first  assistant  or  nurse,  should  be  made,  considering  the  preparation  and  hand-off  of 
instruments  (not  implemented  -  training  scenarios  focused  on  signal  participants  only). 

•  Novices  tended  to  "merely  survive"  the  scenario  by  suturing  the  eyelid  closed  with  any  type  of 
suture  and  without  any  form  of  guidance  as  to  what  the  proper  way  to  do  it  was.  As  sequencing 
of  suture  type  should  correspond  with  the  phase  of  a  scenario  (e.g.  when  to  use  silk  vs.  Vicryl, 
what  size  suture  to  use),  a  suture  dispenser  with  sensors  to  detect  when  a  new  suture  is 
selected  and  its  type/size,  is  a  desirable  feature  (not  implemented  -  later  advice  from  SMEs  was 
to  eliminate  an  integrated  suture  rack). 

•  There  is  a  distinct  pattern  to  the  sequence  of  gestures  an  expert  uses  to  repair  the  laceration 
presented  on  our  eye  portal.  This  should  be  incorporated  into  our  didactic  teaching  material  to 
better  inform  a  novice  performance.  (This  forms  part  of  the  AR  training  system  in  Phase  II) 

•  Occasionally,  experts  performed  maneuvers  outside  of  the  nominal  pattern,  commenting  that 
their  additional  actions  were  part  of  their  "personal  style"  (e.g.  tying  additional,  temporary, 
sutures  in  the  brow  to  hold  the  loose  ends  of  the  suture  neatly  out  of  the  surgical  field).  Use  of 
the  ROI  concept  should  allow  exclusion  of  extraneous  motions  from  analysis. 

•  Sessions  in  2012  were  recorded  with  a  video  camera,  but  the  video  was  not  synced  with  the 
recorded  3D  paths  of  the  user.  Future  system  modification  will  need  to  combine  these  features 
to  ensure  synchronized  video  and  performance  data. 

2. 3. 4. a.  Stand-alone  simulator  structure  revisions 
In  the  Q5  report,  we  presented  a  series  of  concept  drawings  were  prepared  by  one  of  our  student 
interns  pursuing  a  masters  degree  in  design,  that  began  to  address  limitations  of  the  system  used  at  the 
first  user  testing  exercise. 
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Figure  24:  Design  concepts  for  revised  stand-alone  system.  Preliminary  concept  (a)  for  blocking  out 
system  components;  (b-d)  student  concept  designs:  (a)  early  revision  for  instrument  tray  and  integrated 
overhead  lighting  system;  (c-d)  views  of  more  advanced  concepts  including  mobile  arm  to  support 
microscope  head. 


Figure  25:  Evolution  of  panel-based  frames,  including  analysis  of  break-down  for  transportation  in  standard  Pelican 
cases. 

These  designs  integrated  new  means  for  mounting  the  surgical  instruments,  improved  the  positioning  of 
the  magnetic  and  optical  tracking  systems  and  the  display  field  of  the  projector  system,  and  notably 
supported  ease  of  assembly/packing/shipping. 

At  the  close  of  Year  2,  the  structure  had  reached  the  state  shown  below  (middle  panel,  Figure  26),  as 
was  used  at  that  2013  USUHS  Ocular  Trauma  course.  It  is  primarily  assembled  from  laser-cut  acrylic 
panels,  using  a  tab  and  slot  method  [e.g.  Big  Blue  Saw,  2015]  that  has  become  common  with  the 
growing  availability  of  laser  cutters  in  the  commercial  and  private  domain.  The  method  supports 
relatively  easy  prototyping,  assembly  (and  disassembly  for  repair/packing)  requires  only  a  single  screw 
driver,  and  modification  and  replacement  of  components  does  not  require  wholesale  modification  - 
new  slots/holes  can  be  added  to  existing  parts,  as  has  been  done  frequently  during  development.  The 
method  also  supports  the  use  of  threaded  inserts  to  provide  for  secure  connections  between  removable 
components  without  relying  to  threaded  holes  tapped  directly  into  the  plastic  (which  may  wear  or  be 
stripped  through  continued  use). 
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Regarding  the  Year  2  version,  one  of  the  criticisms  noted  was  that  non-sterile  boxes  of  sutures  are  not 
found  within  the  sterile  field  of  a  procedure,  so  the  placement  of  the  suture  boxes  beneath  the 
instrument  holder  rack  was  in  appropriate.  In  Year  3,  the  instrument  rack  was  modified  for  the  2014 
course,  removing  the  suture  boxes  and  lowering  the  instrument  tray  for  easier  access.  In  addition,  as 
will  be  described  in  more  detail  in  sections  2.3.5. b,  2.3.7.C  and  2.3.8,  a  touch-screen  display  mount  and 
integrated  mount  for  the  augmented  reality  microscope  were  added  to  the  system. 

Of  note,  the  head  and  shoulders,  retained  from  the  original  proof-of-concept  system,  were  mounted 
such  that  they  can  be  removed  and  replaced  with  alternate  anatomy,  particularly  that  of  the  later  CT- 
derived  head  form,  described  in  section  2. 3. 5. a. 


Figure  26:  (left)  Early  Year  2  stand,  ca.  1/2013.  (middle)  Year  2  stand,  ca.  5/2013  for  2nd  USUHS  Ocular  Trauma 
Course,  (right)  Year  3  stand  ca.  6/2014,  with  modified  instrument  rack,  integrated  touch-screen  display  and 
augmented  reality  microscope.  The  6/2014  version  allows  for  drop-in  replacement  of  the  head  form  with  the 
more  advanced  version  described  below. 

2.3.4.b.  Force  sensing  strain  gages  for  scissors  and  needle  holders,  characterization  of 
sensing 

In  Year  2,  we  installed  strain  gages  on  all  grasping  (beyond  forceps)  and  cutting  instruments  to  extract 
not  only  open/close  positioning,  but  also  grasp/cutting  forces.  This  enabled  the  detection  not  only  of 
instrument  action,  but  also  interaction  with  the  silicone  tissues,  sutures  and  other  objects,  and  the 
detection  of  their  relative  stiffness  and/or  strength  properties.  Finite  element  analysis  of  the  region 
near  instrument  fulcrums  was  performed  to  show  where  the  ideal  locations  for  strain  gage  placement 
would  occur,  and  for  some  instruments  we  had  professionally  installed  silicon  strain  gages  installed. 
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Figure  27:  FEA  of  scissors  and  needle  holders  showing  optimal  placement  for  strain  gages;  instruments  showing 
design  positions  of  gages  for  force  sensing  and  inactive  gage  for  temperature  compensation.  Top:  curved  Westcott 
scissors.  Middle:  Castroviejo  needle  holder.  Bottom:  Detail  of  scissors  and  needle  holder  with  silicon  strain  gages 
installed 

Detection  of  relative  stiffness  of  grasped  or  cut  objects,  for  example,  allows  discrimination  between 
grasping  of  tissue  vs.  grasping  of  a  suture  needle  vs.  closure  on  empty  space.  This  distinction  can  be 
recognized  by  the  event-based  gesture  recognition  system,  as  the  force-displacement  response  of  the 
instrument  changes. 

In  empty  space,  the  position  sensing  gage  registers  the  user  squeezing  the  instrument  closed,  without  an 
initial  response  from  the  force  sensing  gage,  located  close  to  the  instrument  tip.  Once  the  tips  come 
together,  the  force  sensing  gage  begins  to  respond,  and  the  two  responses  plotted  against  each  other 
generate  a  curve  that  is  related  to  the  instrument's  own  spring  stiffness.  If  an  object  is  grasped,  the 
position  at  which  the  force  response  begins  occurs  earlier  than  for  the  empty  instrument,  giving 
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information  about  approximate  size  of  the  object  grasped.  This  shifts  the  break-point  in  the  force- 
displacement  curve  accordingly.  The  slope  of  the  characteristic  curve  also  changes,  as  the  response  is 
now  a  combination  of  the  stiffness  of  the  instrument  and  the  object  grasped. 


Stiffer  < - >  Softer 


ForcepsTheoretical  Graph 


Figure  28:  Schematic  of  force-displacement  curves  for  use  of  grasping  instruments  in  open  space  and  grasping 
different  stiffness  tissues. 

A  simple  jig  was  designed  to  repeatably  close  the  forceps  (Figure  29).  In  testing,  strain  gage  outputs 
were  collected  for  each  full  turn  of  the  lead  screw.  For  scissors  and  the  needle  holder,  digital  calipers 
were  locked  at  a  series  of  closure  positions. 

In  the  graphs  below  (Figure  30),  taken  using  the  forceps,  the  shifts  in  break  point  and  overall  slope  are 
clearly  shown. 


Figure  29:  Forceps  force-displacement  test  rig. 
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Figure  30:  Normalized  strain  gage  output  vs.  instrument  closure:  red  curve  -  closure  sensor,  blue  curve  -  grasp 
force  sensor.  (Left)  Closing  instrument,  with  tips  coming  together  at  5mm  displacement  of  lead  screw,  with  force 
increasing  as  forceps  are  squeezed  further.  (Center)  Grasping  soft  object,  with  initial  deviation  from  baseline  curve 
at  4mm  displacement,  difference  in  force  slope  with  additional  squeezing.  (Right)  Large,  hard  object,  with  contact 
detected  at  2mm. 

These  same  data  sets  were  collected  for  each  of  the  instruments  used  at  the  2013  Ocular  Trauma  Course 
tests.  Different  instrument  types  show  different  characteristic  responses.  The  needle  holders  are  of  a 
locking  variety,  with  a  spring-loaded  mechanism  that  holds  the  jaws  closed  after  the  user  has  squeezed 
the  instrument  far  enough.  This  results  in  a  momentary  reversal  of  the  force  curve  at  the  locking  point 
and  an  analogous  signal  appears  in  the  closure  position  sensing  response.  Similarly,  the  scissors  show 
almost  no  output  force  in  empty  space  until  complete  closure  is  achieved,  while  force  increases  during 
cutting,  as  the  shearing  forces  generated  by  the  scissors  increase  while  cutting  through  an  object.  In  the 
cutting  case,  as  the  scissors  close  the  cutting  point  shifts  away  from  the  scissor  axis,  increasing  the 
bending  moment  on  the  scissors  (force  x  distance  increases),  and  the  angle  between  the  cutting  edges 
becomes  smaller,  so  a  larger  section  of  the  material  is  being  sheared.  This  results  in  the  non-linear 
increase  in  the  response  observed. 

Described  below,  these  curves  were  built  into  the  gesture  recognition  architecture,  supplementing  the 
earlier,  position-based  (Region  of  Interest)  event  detection  algorithms. 
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Figure  31:  Strain  gage  response  curves:  (upper  left)  0.5mm  jaw  grasping  forceps;  (upper  right)  0.3mm  jaw  grasping 
forceps;  (center  left)  large  jaw  needle  holder;  (center  right)  small  jaw  needle  holder;  (lower  left)  curved  scissors. 

2.3.4.C.  Revision  of  instrument  connector  shells,  stabilization  of  instrument/sensor 
orientation 

In  Q7  we  began  work  to  replace  the  earlier  iteration  of  the  housing  that  connects  the  Ascension  sensor, 
the  strain  gage  amplifiers  and  the  instruments  together.  The  version  used  at  the  2012  USUHS  eye 
trauma  course  data  collection  exercise  was  subject  to  accidental  misalignment  of  the  connector,  which 
prevented  proper  mating  and  distraction  to  the  surgeon/trainee.  The  new  design  includes  spiral  "ramp" 
features  to  guarantee  that  during  plug  insertion,  the  force  of  insertion  between  the  instrument  and  plug 
causes  them  to  rotate  towards  the  correct  connector  alignment.  In  addition,  spherical  features  allow  for 
mating  of  an  instrument  with  the  holding  rack  in  any  orientation  about  the  instrument's  axis.  This 
design  was  retained  through  the  remainder  of  the  development  of  the  system,  with  new  instruments 
having  custom  fittings  designed  to  be  compatible  with  this  form-factor. 
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Figure  32:  New  design  for  instrument/sensor  interconnect  format  with  features  to  make  insertion/removal 
intuitive  and  resistant  to  misalighment.  For  scissor/needle  holder  instruments,  FEA  of  concepts  for  flexural  beams 
for  holding  connector/sensor  housing  aligned  with  long  axis  of  each  instrument.  Not  necessary  for  forceps. 

For  the  scissors  and  needle  holders,  the  pivoting  action  of  the  proximal  ends  allows  the  housing  to  bend 
away  from  the  axis  of  the  instrument,  causing  position  sensing  (in  the  housing)  to  miscalculate  the  tip 
location.  To  prevent  this,  stabilizing  cantilever  flexures  (leaf  springs)  were  designed  into  the  housings  to 
hold  the  scissor/needle  holder  arms  apart.  Early  versions  were  printed  as  part  of  the  housing  using  the 
Polyjet  process  of  our  Objet  rapid  prototyping  system.  This  material  suffers  from  creep  deformation 
when  held  under  continuous  load  (e.g.  needle  holder  locked  closed);  a  solution  was  found  in  the  form  of 
combining  parts  from  both  of  our  rapid  prototyping  systems  -  the  high  resolution  Polyjet  parts  for  the 
rigid  elements  of  the  housing,  with  a  flexible  cantilever  fabricated  using  our  FDM  system,  which  uses  a 
different  plastic  that  appears  to  be  substantially  less  subject  to  creep. 

In  Year  3,  in  response  to  recommendations  from  SMEs  and  expert  users  at  the  USUHS  Ocular  Trauma 
course,  we  acquired  a  punctal  dilator  for  canalicular  repair  scenarios,  Mini  Monoka  stents  (FCI 
Ophthalmics,  Pembroke,  MA  -  see  also  section  2.3.3.b)  for  such  procedures,  and  alternate  cotton  swabs 
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for  use  with  the  suite  of  instruments.  New  designs  for  the  associated  instrument  holders  were  created 
and  fabricated.  The  detailed  design  of  the  holders  for  the  miniature  tying  forceps  and  fabrication  were 
completed  during  Year  3  for  potential  future  use  in  corneo-scleral  repair  scenarios. 

At  the  2014  USUHS  Eye  Trauma  course,  requests  were  made  to  add  straight-jaw  locking  needle  holders 
and  a  standard  cotton  swab  to  the  suite  of  available  instruments.  Over  the  following  months,  new 
instrument  holders  were  designed  for  these  instruments. 


The  final  suite  of  instruments  is  shown  in  Figure  33  (earlier  set  shown  in  Figure  6). 


Figure  33:  Instrument  holder  designs  for  new  devices  (cotton  swab,  punctal  dilator),  earlier  devices  for  globe 
surgery  (straight  and  bent  tying  forceps) 

In  addition,  the  mini-circuit  board  installed  in  the  holders  was  redesigned,  replacing  fixed  gain  and  ID 
resistors  with  potentiometers,  making  adjustment  and  set  up  simpler  for  future  work.  In  earlier  periods, 
we  found  that  the  growing  number  of  instrument  meant  finer  divisions  in  the  ID  signal  to  identify  each 
device.  An  analog  voltage,  ranging  from  2.5V  to  5V  was  the  original  signal  corresponding  to  each 
instrument  -  a  two  surface  mount  resistors,  one  at  approximately  8k  ohms,  the  other  varying  depending 
on  the  device,  were  used  to  set  the  value.  This  restricts  the  available  signal  range,  which  could  in 
principle  be  as  low  as  0V,  given  the  power  and  ground  inputs  to  the  instruments.  The  "miniboards" 
were  redesigned  to  replace  the  fixed  resistors  for  ID  with  potentiometers.  This  caused  the  current 
drawn  by  the  ID  signal  to  be  the  same  for  each  board,  and  allows  outputs  from  0-5V. 

Further,  the  strain  gage  placement  on  each  instrument  and  the  normal  range  of  deformation  of  each 
instrument  is  different,  so  the  instrumentation  amplifier  chip  (the  single  chip  at  the  heart  of  each  board) 
needs  to  have  a  different  gain  setting.  Gain  can  only  be  set  experimentally,  so  installing  and  removing 
SMT  components  repeatedly  is  a  challenge.  Potentiometers  replaced  the  fixed  gain  resistors  for  both 
the  open/close  sensing  portion  of  the  circuit  and  the  force  sensing  portion.  Finally,  the  zero-offset  of 
the  amplifier  also  varies  depending  on  slight  differences  in  the  installation  process  of  the  strain  gages 
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and  on  the  relative  values  of  the  fixed  resistors  in  the  Wheatstone  bridge  circuit  in  use.  Potentiometers 
were  already  in  use  in  the  original  boards,  and  these  were  retained  in  their  positions  in  the  new  version. 


To  access  the  potentiometers,  small  ports  were  added  to  the  holder  shell,  aligned  with  the 
potentiometers  on  both  the  top  and  bottom  of  the  board.  A  jeweler's  slot  screwdriver  is  used  to  make 
adjustments  without  requiring  disassembly  of  the  holder  (a  process  that  also  tended  to  shift  ideal  gain 
and  offset  values).  The  new  boards  were  installed  for  the  cotton  swab,  lacrimal  dilator,  straight  needle 
holder  and  the  straight  and  bent  tying  forceps.  The  remaining  instruments  will  have  replacement 
boards  installed  following  the  current  end  of  the  period  of  performance,  likely  in  conjunction  with 
preparations  for  the  2015  USUHS  Eye  Trauma  course. 


Figure  34:  Adjustment  access  ports  in  latest  instrument  holder  version. 

At  the  2014  USUHS  Eye  Trauma  course,  requests  were  made  to  add  straight-jaw  locking  needle  holders 
and  a  standard  cotton  swab  to  the  suite  of  available  instruments.  Over  the  following  months,  new 
instrument  holders  were  designed  for  these  instruments.  In  addition,  the  holder  for  the  lacrimal  dilator 
previously  acquired  was  completed. 

2.3.4.d.  Polhemus  sensor  evaluation  completion 

As  mentioned  above,  we  evaluated  the  Polhemus  position/orientation  tracking  sensors  available  in  2012 
following  contact  established  with  the  company  at  the  TATRC  booth  at  the  Medicine  Meets  Virtual 
Reality  conference.  We  found  the  increased  cable  flexibility  to  be  a  desirable  feature  as  compared  with 
that  of  the  Ascension  Technologies  system,  however  the  available  sensor  was  too  large  for  the 
ophthalmic  instruments  and  we  observed  additional  noise  added  to  the  strain  gage  sensor  signals. 
Subsequently,  we  held  discussions  with  Ascension  and  evaluated  material  samples  in  regards  to  the 
possibility  of  creating  cabling  with  enhanced  flexibility  and  with  integrated  conductors  to  support  our 
force/grasp  strain  gages,  however  at  that  time  we  chose  not  to  make  modifications  as  the  expense  of 
the  engineering  work  was  not  justified  for  a  relatively  small  improvement  to  our  prototype. 
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Figure  35:  (left)  designs  for  Polhemus  sensor  holders  for  teardrop  sensor  style,  with  standard  and  thin  cable 
implementations,  (right)  Sensor  cabling  implemented  for  physical  testing. 

2.3.5.  “Month  7”  - 

2. 3. 5. a.  Design  of  skeletal  structure,  docking  points,  sensor/actuator  interfaces  with 
ocular  and  craniofacial  trauma  modules. 

2.3.5.a.i.  Creation  of  anatomically  accurate  data  sets  to  support  system  design 
The  basis  for  this  phase  of  the  work  was  the  development  of  a  base  anatomical  data  set,  upon  which  all 
of  the  complex  biological  contours  would  be  based,  and  which  was  modified  as  we  added  the 
engineering  systems  to  actuate  and  sense  from  the  trauma  modules  and  the  main  structure  of  the 
simulator.  We  initially  attempted  to  achieve  this  result  by  identifying  a  high  resolution  CT  or  MR  dataset 
from  the  MGH  patient  data  archives  with  anatomy  including  the  base  of  the  neck  through  the  top  of  the 
head.  When  de-identified  (stripped  of  any  patient  information  outside  of  the  images),  it  would  be 
possible  to  use  this  kind  of  data  as  a  design  tool.  Unfortunately,  after  significant  searching  by  our  SME 
Dr.  Dawson,  by  Drs.  Hugh  Curtin  (Massachusetts  Eye  and  Ear  Infirmary  Chief  of  Radiology)  and  Troulis, 
we  were  unable  to  find  a  suitable  scan.  As  a  result,  we  attempted  to  generate  our  own  data  set  from  CT 
and  MR  imaging  and  physically  molding  a  cadaver  head  in  silicone.  IRB  clearance  was  obtained 
(determined  to  be  not  human  studies  research),  and  biosafety  committee  approval  was  acquired. 


Figure  36:  Process  of  silicone  casting  of  cadaver  head  and  neck  anatomy;  CT  scanning  of  head.  Images  of  MR  scan 
unavailable  due  to  restrictions  on  metallic  objects  in  MR  suite. 
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We  performed  a  trial  run  of  the  molding/imaging  procedure  using  the  first  available  cadaver  sample  that 
was  reasonably  close  to  our  target  demographic  (nominally  male,  average  height,  20-30  years  of  age, 
normal  to  muscular  body  habitus,  no  pathology  of  the  head/neck  region),  however  the  tissue  available 
was  from  an  overweight  individual  in  his  mid-50s.  We  discovered  further  that  the  CT  scanner  was 
unable  to  clearly  distinguish  the  boundary  between  skin  and  the  silicone,  so  segmentation,  even  with  an 
advanced  software  tool  that  we  licensed  on  an  evaluation  basis  (Mimics,  3-Matic,  by  Materialize  NV, 
Leuven,  Belgium),  was  found  not  to  be  possible. 

While  developing  refined  molding/imaging  methods  to  conduct  a  second  set  of  scans  and  waiting  for  a 
more  ideal  candidate  donor,  we  began  to  work  with  a  high  resolution  deidentified  CT  data  set  used 
previously  for  a  neuroendovascular  simulation  system  developed  by  our  group  in  the  mid-2000s  (Luboz 
et  al,  2005).  Use  of  this  data  set  was  determined  to  be  Not  Human  Use.  While  this  data  set  did  not  have 
matching  MR  imagery  or  a  physical  mold,  it  was  created  with  thin  slice  spacing.  Skin  and  bone  contours 
and  a  number  of  the  soft  tissue  elements  could  be  segmented  directly  while  final  treatments,  such  as 
skin  textures,  were  included  after  3D  printing  of  the  relevant  sections  of  the  reconstructed  head  form. 
Figure  37  shows  segmented  skeletal  features  and  skin  surface,  and  preliminary  imagery  of  how  the  face 
sections  would  be  segmented  so  that  elements  could  be  removed  and  replaced.  The  precise  division  of 
facial/eye  trauma  zones  continued  to  evolve  as  we  designed  in  the  necessary  mobility  and  functional 
features.  Figure  39  shows  sections  of  the  segmented  anatomy  reproduced  in  rigid  plastic  and  a  semi- 
flexible  rubber,  which  we  created  to  explore  the  capabilities  of  the  Objet  3D  fabrication  system,  which 
we  acquired  through  this  program  and  which  came  online  in  early  2012. 

In  addition  to  the  structural  design  elements  described  above,  we  worked  toward  subdividing  tissue 
types  and  experimenting  with  molding  techniques  to  generate  the  various  portals.  Towards  the  end  of 
Year  1,  to  support  compatibility  with  COTS  components  and  commercial  mannequin  systems,  we 
examined  elements  such  as  the  Laerdal  airway  model  to  use  as  a  platform  for  testing  the  sensors  prior 
to  developing  our  own  versions  with  improved  anatomy,  but  with  common  connection  points. 


Figure  37:  Segmented  CT  images  from  prior  simulator  project;  reassembled  surface  models  of  skin,  bone,  eye 
globes;  rendering  of  reconstruction  with  flesh  tones  applied.  Skin  textures  would  be  applied  during  creation  of 
physical  molds 
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Figure  38:  Early  concept  sketches  and  later  CAD  models  of  methods  for  dividing  structural  elements  of  the 
head/skull  to  permit  creation  of  modular  injury  components  and  recent  scans  and  combinations  of  COTS  and 
custom  components. 


Figure  39:  Early  concept  for  division  of  head  elements  using  Solidworks  CAD  tools;  rapid  prototyped  rubber  and 
hard  plastic  sections  created  from  CAD  models. 

2.3.5.a.ii.  Eye  motion  mechanism 

Eye  globe  motion,  both  passive  and  controlled,  was  an  important  improvement  over  the  fixed  globe  of 
the  proof-of-principle  system.  For  the  unconscious  patient,  the  globe  should  be  movable  when 
manipulated  with  surgical  instruments,  and  for  the  conscious  patient,  the  globe  should  allow  for 
controlled  motion.  For  conditions  including  retrobulbar  hemorrhage,  the  eye  ceases  to  be  mobile  due 
to  the  increased  pressure;  when  relieved,  some  of  the  motion  returns.  In  addition,  retrobulbar 
hemorrhage  causes  the  globe  to  be  pushed  forwards  (proptosis),  so  an  additional  mechanism  needed  to 
be  included  to  generate  this  motion.  The  measurement  of  intraocular  pressure  or  force  loading  onto 
the  globe  must  also  be  measured  to  detect  (a)  when  retrobulbar  hemorrhage  is  successfully  treated 
through  lateral  canthotomy  and  cantholysis  and  (b)  when  eye  shields  or  other  coverings  are  applied 
around  the  eye,  to  verify  that  inappropriate  pressures  are  not  applied  to  the  globe  during  the  treatment 
of  nearby  bleeding. 

In  year  1,  we  developed  initial  prototypes  to  investigate  motion  systems  are  shown  below.  They  used  a 
pivoting  joint  at  the  center  of  the  globe  and  cable  and  linkage  mechanisms  to  link  to  actuators.  The 
actuation  and  transmission  systems  were  envisioned  to  reside  within  the  permanent  structure  of  the 
mannequin  head,  with  interchangeable  eye  globe  modules  to  represent  different  conditions  (e.g. 
lacerated  globe,  undamaged  globe).  The  actuation  mechanisms  were  designed  to  be  "backdrivable",  so 
that  external  surgical  manipulation  would  move  the  eye. 
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Figure  40:  Concepts  and  prototypes  for  eye  motion  mechanism.  Includes  early  concept  test  of  mechanical,  petal- 
based  pupil  contraction  mechanism. 


Figure  41:  Eye  motion  module  for  pitch,  yaw  and  proptosis,  with  backdriveability  in  globe  motion.  Sizing  of  internal 
components  relative  to  skull  model. 

In  year  2  building  on  earlier  design  concepts,  we  designed  and  prototyped  a  mechanism  that  includes  all 
of  the  globe  motion  features  to  be  included  in  the  eye  modules:  eye  pitch  and  yaw,  proptosis, 
backdriveability  in  pitch  and  yaw  (to  allow  the  surgeon  to  manipulate  the  eye)  and  sensing  of  applied 
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force  to  detect  pressure/load  applied  to  the  globe  and  the  reduction  of  force  on  the  eye  expected  on 
successful  canthotomy/cantholysis. 

The  mechanism  shown  in  Figure  41  was  the  first  iteration  that  supported  the  mechanical  motions  and 
compliance  features  analogous  to  normal  eye  motions,  and  relative  sizes  of  the  components  in 
reference  to  our  CT-derived  skull  model. 

To  allow  for  attachment  of  replaceable  eye  globe  models  and  provide  fluid  and  signal  connectivity,  we 
included  fluid  channels  that  protrude  into  the  space  of  the  globe.  These  serve  the  dual  purpose  of 
providing  mechanical  connection  through  a  tight  friction  fit.  The  fluid  channels  connect  the  globe  to  a 
blood  simulant  supply  for  simulation  of  bleeding  and/or  hemorrhagic  chemosis  -  bleeding  behind  the 
conjunctiva  creating  a  red  zone  around  the  cornea,  one  of  the  common  signs  of  retrobulbar 
hemorrhage.  They  would  also  provide  channels  for  fluid  for  testing  of  suture  water-tightness  and  future 
capabilities  such  as  control  of  pupil  contraction.  Figure  42  shows  modifications,  including  reduction  in 
thickness  (to  accommodate  large  angular  motion  without  collision  between  fluid  connections  and  the 
supporting  post),  the  fluid  channels  with  connection  points  front  and  back  and  corrections  to  the  central 
spherical  joint  design  from  the  earlier  version.  It  also  integrated  a  first  test  at  a  revision  to  the  elastic 
member  connection,  replacing  the  latex  rubber  bands  with  neoprene  o-ring  material  for  improved 
reliability  and  lifetime  vs.  latex  rubber.  The  original  hook/loop  attachment  mechanism  was  replaced 
with  a  version  no  longer  subject  to  the  possibility  of  the  elastic  member  becoming  dislodged,  as  set 
screws  maintain  the  connections. 


Figure  42:  base  plate  modifications,  rapid  prototyped  test  versions  of  plate  and  globe  showing  plate  and  globe 
segment  separately  and  assembled.  Features  include  bosses  for  neoprene  cords  to  replace  latex  rubber  bands. 
Modified  elastic  transmission  components:  o-ring  cord  threaded  through  servo  arm,  initial  baseplate  attachment. 

During  Q6,  examination  of  sections  through  the  model  of  the  skull  was  done  to  verify  accommodation  of 
the  mechanism  within  the  skull  contour  while  maintaining  a  reasonable  thickness  of  skull  structure 
external  to  the  motion  mechanism.  The  initial  orthogonal  arrangements  of  eye  motion  mechanisms 
would  violate  the  outer  contour  of  the  skull,  so  they  were  angled  inwards  and  the  servo/cord 
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arrangements  and  contour  of  the  globe  base  plates  modified  to  retain  the  normal  forward  looking 
orientation  of  the  globe. 


Figure  43:  Sectioning  of  skull  with  prototype  motion  mechanism  showing  excursion  of  elastic  cords  from  surface  of 
skull  in  zygomatic/sphenoid  region. 

A  proptosis  mechanism  designed  early  in  Q5  reached  a  functional  test  model  by  the  end  of  the  quarter, 
demonstrating  a  range  of  motion  of  approximately  10mm,  more  than  sufficient  to  generate  the 
protrusion  of  the  globe  necessary  to  indicate  retrobulbar  hemorrhage.  Shown  below  is  proof  of 
principle  design  and  the  sliding  carriage  shown  in  the  extended  configuration. 


Figure  44:  Proptosis  mechanism  (separate  from  pitch/yaw  components) 

Further  revisions  to  the  design  included  selection  of  smaller  servo  motors,  unification  of  the  structure 
into  fewer  separate  components,  and  flipping  the  orientation  of  one  servo  of  each  pair  to  reduce  size  of 
the  mechanism  and  prevent  collision  between  the  control  arms.  These  changes  enabled  the  mechanism 
to  be  rotated  15  degrees  from  the  original  position  to  better  align  with  the  eye  socket  and  preserve 
orbital  wall  thickness.  A  prototype  of  the  compact  version  of  the  eye  motion  system  was  built,  shown 
below. 
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Figure  45:  comparison  of  initial  and  modified  orientations  of  eye  motion  mechanisms,  preserving  orbital  wall 
thickness  and  reducing  overall  mechanism  size.  Assembled,  compact  motion  module. 

In  Q6,  the  final  main  actuated  component  of  the  eye  motion  system,  the  blinking  mechanism  was 
prototyped,  so  that  the  mannequin-mounted  version  of  the  system  could  provide  a  sense  of  presence 
and  indication  of  level  of  consciousness,  and  help  to  provide  indications  of  eye  trauma;  blinking  would 
be  prevented,  for  example,  during  a  retrobulbar  hemorrhage  scenario. 

The  eyelid  is  made  of  silicone  soft  layers  (see  section  2.3.5.a.iii  for  eye/face  module  design),  mounted  to 
an  orbital  rim  structure.  A  silicone/mesh  protrusion  of  the  eyelid,  corresponding  roughly  with  the 
levator  and  Muller's  muscles,  passes  through  a  gap  created  in  the  upper  part  of  the  orbital  rim  of  the 
portal.  This  protrusion  will  links  with  a  retraction  mechanism,  part  of  the  eye  motion  assembly.  Shown 
below  is  Q6  iteration  of  the  design  of  the  bony  components  of  the  system,  with  protrusion  slots. 
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Figure  46:  Orbital  rim  installed  in  skull,  with  slot  for  "levator  muscle"  protrusion  of  eye  lid;  orbital  rim  removed 
with  slot  for  access  to  blinking  mechanism  behind  forehead. 


The  blinking  mechanism  is  a  servo-actuated  crank-slider,  with  an  end-hook  which  will  mate  with  the  eye 
lid  protrusion.  A  crank-slider  linkage  is  necessary  to  orient  the  pulling  direction  of  the  eyelid  and  to 
avoid  collision  with  the  internal  wall  in  the  skull. 


Figure  47:  Blink  motion  eyelid  retraction  mechanism.  Skull  segment  with  motion  mechanism  for  testing  of 
assembly  and  fit.  Also  shown  is  partially  seated  bony  orbital  rim  which  will  support  the  trauma  portal  soft  tissue. 


Also  in  Q6,  the  contralateral  portion  of  the  system  was  designed.  A  reflection  of  the  mechanism  across 
the  sagittal  plane  results  in  collision  between  the  servo  control  arms  of  the  posterior  servo.  Instead, 
keeping  the  same  servo  mounting  frame  and  rotating  its  placement  prevents  collisions  and  reduces  part 
count.  This  can  be  seen  in  the  detail  of  Figure  48. 


The  slide  mechanisms  for  proptosis  were  lengthened,  their  anterior  ends  modified  to  support  the  angled 
orientation  of  the  mechanism,  and  the  main  frame  that  supports  the  assembly  widened  with  mounting 
features  for  attachment  to  the  skull.  To  further  unify  the  design,  the  same,  smaller  servos  used  for  eye 
motion  were  used  for  the  proptosis  motion.  This  design  decision  simplifies  the  bill  of  materials,  also 
reduces  mechanism  size  and  simplifies  software  control  of  the  system. 


Two  additional  servos  are  shown,  used  for  the  blinking  mechanism. 
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Figure  48:  Bilateral  eye  motion  mechanism  with  servo  configuration  maintained  left/right  vs.  reflected.. 


Figure  49:  Partitioning  of  skull;  skull  component  with  motion  cavity  shown;  assembled  skull  design  with 
mechanism. 

The  bilateral  assembly  was  connected  to  our  drive  circuitry  for  testing  and  debugging  of  the  code, 
mainly  to  ensure  that  all  of  the  independent  and  linked  motions  behave  correctly  -  blinking  of  both  eyes 
occurring  in  the  same  direction  (the  servos  are  arranged  opposite  to  each  other,  so  inverse  command 
signals  must  be  sent  to  them  for  the  output  shafts  to  generate  the  same  blinking  motions). 


Figure  50:  CAD  rendering  of  bilateral  version  of  eye  motion  mechanisms  and  initial  testing  of  servo  drives  of 
pitch/yaw  and  propotosis  actions. 
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2.3.5.a.i.  Eye  motion  kinematics 

Because  the  position  of  the  elastic  cables  would  physically  interfere  with  rigid  components  of  the  system 
if  the  servos  were  oriented  at  90  degrees  to  each  other,  the  servos  are  rotated  50  degrees  away  from 
parallel,  or  +/-  25  degrees  away  from  the  vertical  y-axis  shown  in  Figure  51.  In  this  configuration, 
rotation  of  the  globe  requires  motions  of  both  servos. 


The  eye  mechanism  can  rotate  in  both  the  yaw  (a)  and  pitch  ((3)  directions,  under  the  control  of  the  two 
independent  servos,  with  rotations  0i  and  02. 


Figure  51:  Eye  motion  polar  coordinate  system.  Globe  motion  control  is  defined  about  the  X  and  Y  axes,  while  the 
servos  rotate  at  offset  axes,  with  motions  0i  and  02. 

To  make  the  calculations  more  general,  the  servo  offset  angle  is  defined  as  y  and  the  command  positions 
sent  to  the  servos  for  a  given  eye  are  determined  by: 


0i 

$2 


tan 


(sin  a  cos  y  +  cos  a  sin  /?  sin  y) 


tan 


-l 


cos  a  cos  /? 
and 

(sin  a  cos  y  —  cos  a  sin  /?  sin  y) 


cos  a  cos  /? 


These  equations  were  implemented  in  the  motion  controller  for  testing  the  eye  motion  mechanism.  A 
basic  point-and-click  interface  was  written  to  control,  eye  orientation,  proptosis  position  and  motion 
speed. 
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Using  an  Arduino  Duemilanove  microcontroller  and  a  graphical  user  interface  written  in  Processing, 
motion  control  of  the  mechanism  was  achieved.  This  formed  the  basis  of  a  later,  more  comprehensive 
controller,  described  below.  Examples  of  control  entry  and  corresponding  eye  mechanism  positions  are 
shown. 


proptosis:  +14mm 


Figure  52:  Processing  control  panel  for  pitch/yaw/proptosis  showing  motions  of  mechanism  corresponding  with 
graphically  entered  target  positions. 

During  Q7,  we  described  a  move  from  the  Arduino  2009  platform  for  interfacing  with  the  eye/head/neck 
control  systems  to  the  Arduino  Mega  2560.  The  alternate  board  supports  40  more  digital  I/O  lines  and 
10  additional  analog  input  channels.  Based  on  the  restrictions  of  the  central  microprocessor  and  our 
needs  for  controlling  multiple  functions,  we  replaced  a  stock  interface  program  available  for  the  Arduino 
with  one  of  our  own,  which  optimizes  the  available  I/O  channels  for  our  use.  The  board  was  configured 
to  run  8  different  latching  valves,  12  servos  (primarily  eye  mechanism  motion),  I2C  communications  to 
the  digital-to-analog  modules  for  use  with  the  electronic  regulators  we  acquired  earlier,  one  or  two 
stepper  motors,  two  quadrature  encoders,  and  unassigned  I/O  for  10  PWM  or  digital  I/O  lines  (e.g. 
pump  control,  single  acting  valves),  three  serial  communication  ports,  and  all  16  analog  input  lines. 


To  test  this  system  and  provide  control  for  the  existing  hardware  elements,  we  prepared  the  GUI 
currently  being  used  with  the  eye  motion  platform  described  above  and  the  latching  valves  on  the  air 
muscle  test  bed  developed  earlier  (see  section  2.3.5.a.v). 
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Figure  53:Comprehensive  development  graphical  user  interface,  including  pneumatic  controls  (upper  left), 
proptosis  position  and  speed  for  left  and  right  eye  (upper  right),  blinking  servo  position  and  speed  and  whether 
left/right  eyes  are  linked  (middle  right),  and  independent  or  linked  eye  orientation  control 

The  control  circuitry  was  unified  as  well,  shifting  from  an  initial  version  with  discrete  transistors  (see 
Figure  54)  to  a  smaller  number  of  ICs.  A  Bluetooth  module  was  added  to  the  circuit,  which  would  allow 
the  Instructor  Tablet  computer  associated  with  the  mannequin  (Laerdal  SimMan  Essential,  see 
mannequin  acquisition  and  modification  sections  below)  to  control  the  new  components  wirelessly. 
When  connected  to  the  mannequin's  power  supply,  we  achieved  a  state  where  the  new  components 
require  no  external  wired  connections. 

Testing  of  the  valves  in  late  Q7  showed  that  the  12V  mannequin  supply,  when  run  through  the  earlier 
discrete  transistor  circuit  or  the  new  version,  did  not  produce  sufficiently  high  voltage  at  the  12V-rated 
valve  inputs.  Voltage  drops  across  the  transistors  result  in  insufficient  voltage  at  the  valves,  so  we  have 
observed  that  some  of  the  valves  are  not  reliably  actuated.  To  solve  this  issue,  we  ordered,  installed 
and  tested  a  DC/DC  converter,  which  boosted  the  12V  supply  from  the  mannequin  to  15V  output  of  the 
converter.  This  comes  at  the  cost  of  a  higher  current  draw  from  the  mannequin,  so  we  were  limited  to 
running  two  valves  at  a  time  at  most  (mannequin  current  limit  is  1A).  Since  the  valves  would  generally 
be  activated  one  or  two  at  a  time  during  normal  operation,  this  is  not  expected  to  be  issue  -  only  during 
initialization  of  the  system  would  numerous  valves  be  activated  in  short  order,  but  this  can  be  done 
sequentially  rather  than  all  in  parallel,  and  since  the  time  to  pulse  the  valves  to  set  their  state  is  under 
100ms,  this  did  not  present  problems. 
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Figure  54:  Protoboard  for  eye  servo  and  neck  pneumatic  valve  control 
2.3.5.a.ii.  Pupil  contraction 

Demonstrating  responsiveness  to  light  or  to  level  of  consciousness  or  to  certain  pathology,  pupil 
contraction/dilation  was  felt  to  be  an  important  element  for  the  eye  modules,  especially  in  scenarios 
where  the  patient  would  be  conscious.  In  Year  1,  we  reviewed  and  tested  a  variety  of  technologies  to 
generate  such  motion,  via  mechanisms  that  could  fit  within  a  portion  of  the  space  inside  the  globe.  For 
the  suite  of  eye  modules  envisioned,  surgeons  would  not  explore  or  intervene  inside  the  sclera,  so  the 
posterior  chamber  can  be  substantially  used  for  actuation  or  sensing  elements. 

The  first  test  pupil  used  ferrofluids  in  the  presence  of  an  electromagnetic  field.  We  attempted  to  create 
a  small,  closed  reservoir  of  ferrofluid,  which  would  expand  and  contract  when  the  magnetic  field 
strength  of  the  electromagnet  was  varied.  We  found  that  the  drive  current  passing  through  the 
electromagnet  and  size  of  the  wire  coil  required  to  generate  a  useful  response  were  both  too  large  to  be 
practical  for  installation  within  the  space  of  the  eye  globe.  In  addition,  creation  of  a  varying  magnetic 
field  in  the  vicinity  of  the  magnetic  position  tracking  system  would  create  an  unacceptable  and 
unpredictable  distortion  of  the  position  measurements. 

Next  we  examined  the  use  of  Nitinol  shape  memory  alloy  (SMA)  actuation,  which  does  not  create  a 
significant  magnetic  field  during  use,  and  has  a  very  small  volume  to  actuation  force  ratio.  The  SMA  was 
coupled  to  a  series  of  filaments  that  pulled  the  inner  margin  of  a  silicone  iris  inwards,  with  the  outer 
margin  anchored  and  providing  a  restoring  force.  The  contractile  filament  concept  was  found  to  be 
reasonably  successful,  though  was  found  to  be  complicated  to  assemble.  The  main  challenge  was  that 
when  SMA  is  actuated,  it  contracts  by  less  than  5%  of  its  rest  length;  to  achieve  a  reduction  in  pupil 
diameter  from  8mm  to  4mm  (smaller  than  the  normal  human  range  of  pupil  sizes)  a  2mm  contraction  of 
the  SMA  would  be  required,  necessitating  a  total  length  of  the  SMA  of  over  40mm,  larger  than  the 
diameter  of  the  eye  globe  (typically  about  25mm). 
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Figure  55:  Examples  of  test  platforms  for  evaluating  tensile  filament  actuation  of  elastic  iris  using  shape  memory 
alloy  actuators. 

A  further  approach,  to  keep  the  actuator  size  approximately  equal  to  that  of  the  iris,  we  created  molding 
techniques  to  make  a  toroidal  "balloon",  which  could  be  inflated  or  evacuated  with  air  or  fluid  pressure. 
With  the  outer  margin  of  the  torus  restrained,  the  inner  diameter  is  reduced  when  the  torus  is  inflated. 
The  example  shown  below  makes  use  of  plain,  unpigmented  silicone  however  any  color  could  be  used 
for  cosmetic  purposes. 


Figure  56:  Sequence  of  inflation  of  toroidal  "iris"  to  cause  contraction  of  pupil. 

We  also  prototyped  interleaved  iris  "petals",  somewhat  analogous  to  a  camera  aperture.  With  the  3D 
fabrication  system  acquired  for  this  program,  we  were  able  to  create  complex  flexural  joints  to  allow  for 
a  natural  contractile  motion  with  a  relatively  circular  pupil  perimeter.  A  pair  of  filaments  wraps  around 
all  of  the  petals,  and  when  tightened,  through  the  action  of  a  small  motor,  causes  the  petals  to  converge 
towards  the  midpoint  of  the  pupil.  We  conducted  experiments  to  look  at  the  effects  of  a  motor  in 
operation  near  the  magnetic  position  trackers.  Provided  that  motors  are  mounted  towards  the 
posterior  portion  of  the  head,  we  found  that  they  do  not  generate  noticeable  distortion  in  instrument 
positioning.  The  flexure  assembly  was  designed  to  fit  within  the  volume  of  the  eye  globe,  and  continues 
to  remain  under  consideration  for  use  in  future  eye  modules. 
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Figure  57:  Two  iterations  of  contracting  petal  concept  for  pupil  contraction  mechanism  with  testbed  motor  driven 
system.  Transmission  makes  use  of  twist-actuator,  shortening  the  tensile  cable  as  it  is  twisted  by  the  stepper 
motor. 

Lastly,  we  contacted  the  E-Ink  company  (E-Ink  Corp.,  Billerica,  MA),  which  makes  electroactive  displays 
that  would  have  very  good  visibility  under  daylight  or  other  conditions,  would  have  low  power 
consumption  and  very  small  volume.  Unfortunately  their  response  was  that  the  non-recurring 
engineering  costs  could  only  be  justified  if  the  annual  minimum  order  quantity  is  in  the  100,000+  range. 

2.3.5.a.iii.  Periorbital  tissue  modules 

Year  2  saw  significant  advances  over  the  original  proof-of-concept  and  Year  1  versions  of  the  updated 
eyelid  modules,  which  included  a  simple  tarsal  plate  structure.  Much  experimentation  was  done  to 
explore  the  potential  range  of  anatomical  features  that  could  be  included  in  the  anatomical  modules,  a 
subset  of  which  were  implemented  in  the  versions  presented  at  the  USUHS  courses,  and  all  of  which 
remain  available  for  future  implementation  in  potential  commercial  versions  of  the  system  or  its 
subsystems. 

During  Q5,  we  began  to  add  silastic  tubing  into  the  margin  of  the  eyelid  model  to  provide  the  anatomy 
for  performing  canalicular  repair,  for  more  advanced  eye  lid  laceration  scenarios  that  the  initial  full¬ 
thickness  marginal  laceration.  Dr.  Mazzoli  provided  advice  and  detail  on  the  relative  sizes  of  the 
canaliculi  and  the  punctum,  the  small  opening  that  drains  tears  from  the  eye. 

In  addition,  we  began  to  generate  CT-derived  skin  surface  models  to  generate  new  eye  lid  structures,  vs. 
our  earlier  Phase  I  system  that  used  the  life-cast  original.  Shown  below  are  surfaced  CT  data,  CAD 
models  of  elements  of  the  mold,  partially  molded  lid  structures,  with  the  first  mold  lid  removed  prior  to 
installation  of  the  second  lid  with  the  cavity  for  the  tarsal  plate,  and  the  front  and  back  views  of  the  new 
lid  structures. 
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Figure  58:  Eyelid  based  on  CT-derived  data:  (top  row)  Mimics  software  view  of  anatomy,  CAD  model  of  mold, 
partially  completed  eye  lid,  prior  to  over-molding  of  tarsal  plate,  (bottom  row)  Front  and  back  views  of  overmolded 
portal  structure,  CAD  model  with  laceration,  physical  test  model  with  defined  laceration  molded-in. 

We  tested  a  series  of  models  in  which  the  "laceration"  would  be  designed  directly  into  the  3D  printed 
molds,  so  that  differences  between  trauma  modules  due  to  artistic  variation  or  between  instructors 
creating  their  own  lacerations  would  not  occur. 

Using  the  components  that  were  produced  at  that  time,  we  also  performed  tests  to  determine  how  to 
create  molded  eye  lids  with  multiple  layers  that  would  be  compatible  with  the  blinking  mechanism 
described  in  the  earlier  sections.  The  initial  part  of  the  test  was  to  create  incisions  in  the  lids  and  attach 
threads  to  examine  the  likely  motion  during  retraction.  These  tests  suggest  that  an  approach  to  the  soft 
tissue  components  is  to  incise  the  skin  layer,  seat  the  molded  soft  tissue  into  a  second  mold  with  the  lid 
in  a  hyper  extended  position  and  mold  in  additional  silicone  to  create  the  redundant  tissue  that  Dr. 
Mazzoli  described  for  us  in  earlier  conversations.  This  redundant  tissue  rolls  over  during  the  eye  lid 
opening  process,  creating  the  fold  of  the  eye  lid.  Using  the  3D  models  extracted  from  the  CT  scans  using 
Mimics  and  3-matic  an  inner  layer  was  designed  that  extends  through  a  cutout  above  the  upper  eyelid 
and  an  extruded  feature  extending  upward  from  the  tarsal  plate.  Initial  testing  of  a  modified  module 
demonstrating  blinking  under  servo  control  is  described  in  2.3.9. b. 
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Figure  59:  Blinking  soft  tissue  test  object;  design  for  eye  lid  with  levator  muscle  structure;  test  version  installed  in 
skull  section,  with  orbital  rim  and  assembled  soft  tissue  layers. 

During  Q7,  work  on  moving  the  lacrimal  duct  structures  forward,  necessary  to  complete  development  of 
an  eye  module  for  lacerated  lid  with  canalicular  repair. 

The  lacrimal  assembly  in  this  iteration  consists  of  the  canalicular  ducts  and  the  nasolacrimal  sac/duct. 
With  the  ducts  running  through  the  margin  of  the  upper  and  lower  eyelids,  the  module  mold  includes 
alignment  slots  for  the  canalicular/tarsal  assembly.  The  ducts  consist  of  silastic  tubing  of  an  appropriate 
diameter,  bonded  and  sealed  with  silicone  glue.  The  assembly  is  then  attached  to  the  tarsal  plate  using 
nylon  string.  The  whole  assembly  is  then  seated  in  the  revised  mold  using  the  new  alignment  slots. 
Sprues,  normally  a  by-product  of  the  molding  process  that  is  discarded,  are  preserved  to  serve  as 
alignments  pegs.  The  mold  revision  ensures  proper  placement  of  the  assembly  in  the  mold  prior  to 
injection.  The  entire  assembly  is  then  encapsulated  by  silicone  during  the  casting  of  the  portal,  resulting 
in  a  lifelike,  multilayered  object. 
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Figure  60:(Left)  eye  tarsal  plate,  with  attached  lacrimal,  nasolacrimal  duct  tubing,  reinforcing  mesh  (as  per  previous 
versions  of  the  portal)  (Right)  Mold  for  eyelid  portal,  with  tarsal  plate,  ducts  in  place  for  over-molding  with  softer 
durometer  silicone  for  eyelid/skin  structures.  Sprues  from  tarsal  plate  molding  process  hold  the  plate  and  duct 
structure  in  place  for  overmolding  process. 

During  Q8  we  implemented  and  improved  upon  the  canalicular  structures.  The  overall  size  and  position 
of  the  silastic  tube  was  modified  to  more  accurately  reflect  the  anatomical  structures.  Small  grooves, 
similar  to  the  ones  present  in  an  actual  lateral  canthal  ligament,  have  been  replicated  in  the  module  and 
securely  hold  the  canaliculus  in  place. 

In  Year  3,  after  review  by  the  SMEs,  it  was  determined  that  it  would  be  extremely  difficult  and  not  useful 
to  excise  the  canaliculus  from  the  surrounding  tissue  in  an  actual  surgery,  and  that  there  is  no  reason  for 
the  simulator  to  be  able  to  do  this.  Instead,  the  canaliculus  is  contained  within  the  muscle  and  tissue 
that  makes  up  the  medial  canthal  tendon.  Therefore,  the  simulated  canaliculus  need  only  be  a  channel 
that  can  be  located  by  identifying  its  white  colored  surface  and  can  be  intubated.  When  lubricated  with 
a  small  amount  of  silicone  oil,  the  part  pictured  below  fulfills  this  requirement. 


Canalicular  channels 


Figure  61,  Canalicular  channels 

In  Year  2  we  made  a  substantial  redesign  for  the  tarsal  plate  part  for  use  in  a  second-generation  module 
that  we  showed  to  our  SME  at  the  2013  USUHS  Ocular  Training  Course.  Unlike  the  previous  generation 
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tarsal  plate  part,  a  one-piece,  flat  band  that  resembles  a  simplified  version  of  the  anatomical  structure, 
the  second-generation  tarsal  plate  more  closely  resembles  the  real  anatomy.  It  includes  a  well-defined 
medial  canthal  ligament  which  contains  the  aforementioned  canalicular  grooves  in  the  superior  and 
inferious  crus,  an  anatomically  accurate  lateral  canthal  ligament,  Whitnal's  ligament,  Lockwood's 
ligament,  and  the  upper  and  lower  tarsal  plates. 
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Figure  62:  (Left)  Textbook  anatomy  of  tarsal  plates,  periorbital  ligaments  used  as  reference  for  advanced  module 
design.  (Right)  Molds  and  cast  version  of  new  anatomy,  fabricated  in  stiffer  silicone  than  surrounding  eyelid  tissue. 

All  of  these  structures  are  cast  as  a  single  part.  The  part  aids  in  providing  a  realistic  surgical  experience, 
strengthens  the  overall  structure  by  providing  more  surface  area  for  the  module  to  bond  with,  and 
provides  a  robust  framework  for  the  attachment  of  the  canalicular  tubes.  Furthermore,  the  lateral 
canthal  ligament  and  Lockwood's  ligament  have  been  outfitted  with  wires  to  serve  as  cut  detectors 
during  the  proptosis  procedure.  During  this  procedure,  a  surgeon  is  required  to  cut  through  the  lateral 
canthal  ligaments,  bisecting  it  until  they  reach  the  Lockwood's  ligament.  They  then  bisect  the 
Lockwood's  ligament  until  pressure  on  the  orbit  is  released.  The  embedded  wires  indicate  whether 
these  incisions  have  been  properly  executed.  The  electronics  for  this  detection  circuit  are  deliberately 
simple;  the  wires  through  the  ligaments  tie  a  digital  input  channel  on  the  Arduino  microcontroller  (or 
other  later  system)  to  ground  (also  providing  electrical  safety  as  there  is  little  current  passing  through 
the  wires  at  any  time).  When  the  wires  are  cut,  the  ground  connection  is  lost  and  a  pull-up  resistor 
internal  to  the  microcontroller  raises  the  measured  voltage,  signaling  the  incision. 
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Figure  63:  (upper  row)  Front  and  rear  views  of  new-version  trauma  module  (uninjured  version).  Soft,  pigmented 
silicone  and  hard,  clear  version  for  illustration.  Internal  structures  include  tarsal  plate  with  ligaments,  periorbital 
fat  deposits,  arterial  vasculature,  canaliculi  and  naso-lacrimal  duct.  Canthal  ligaments  have  fine  wire  embedded  in 
them  (visible  in  rear-view,  bare  wires  and  attached  to  temporary  connector),  arterial  system  has  inlet  and  outlet 
ports  for  blood  simulant  flow  and  external  pressure  sensing,  (lower  row)  Module  connected  to  microprocessor 
showing  illuminated  LEDs  that  indicate  un-cut  anatomy. 
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Figure  64:  (Left)  Textbook  anatomy  showing  fat  deposits,  lacrimal  gland.  (Right)  Intermediate  step  in  advanced 
module  casting,  showing  placement  of  periorbital  fat  deposits. 
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The  module  was  further  augmented  by  the  inclusion  of  a  layer  of  periorbital  fat  and  a  lacrimal  gland 
(Figure  64).  The  fat  adds  to  the  overall  realism  of  the  module,  creating  a  more  lifelike  feel.  Additionally, 
the  fat  could  be  used  to  simulate  a  subconjuctival  prolapse.  It  is  possible  to  view  the  fat  deposits  as  well 
as  the  lacrimal  gland  by  inverting  the  eyelid  with  a  forceps.  These  structures  add  a  level  of  realism  that 
with  previously  unavailable. 

We  further  augmented  the  module  through  the  inclusion  of  a  functional  arterial  system  (Figure  65).  An 
analog  of  the  authentic  orbital  arterial  system  was  designed  and  created  for  the  purpose  of  providing 
the  option  of  a  lifelike  bleeding  scenario.  The  arteries  can  be  filled  with  blood  simulant  fed  from  an  IV 
bag  or  active  pump  system.  Pressure  of  the  arterial  system,  monitored  by  an  external  pressure  sensor, 
drops  when  an  artery  is  cut,  and  hemorrhage  of  the  simulant  occurs.  This  data  can  be  incorporated  into 
the  model.  We  can  also  detect  when  a  hemorrhage  is  stopped  by  the  user,  opening  up  the  possibility  for 
additional  future  scenarios. 


Figure  65:(Left)  Textbook  anatomy  showing  arterial  network.  (Center)  Inlay  of  wax  vessels  to  preserve  vessel 
lumens  -  wax  is  melted/dissolved  out  after  silicone  cures  to  produce  empty  vessel  lumens.  (Right)  fully  assembled 
advanced  module,  with  inlet  and  outlet  taps  for  bleeding  from  eyelid  laceration  module. 

In  Q9,  the  skin  layer  of  the  trauma  module  was  modified  to  include  previously  lacking  landmarks 
normally  associated  with  the  eyelid,  notably:  the  bulge  in  the  lower  eyelid  caused  by  the  "bunching"  of 
skin  when  the  eye  is  open;  a  prominent  fold  in  the  upper  eyelid;  the  Meibomian  glands;  and  the  lacrimal 
caruncle  -  the  crease  of  skin  at  the  medial  angle  where  the  eyelid  terminates.  These  are  prominent 
geographic  landmarks  of  the  eyelid  which  were  missing  from  our  model.  Due  to  the  extensive 
modification  of  component  parts  of  the  module,  it  was  necessary  to  make  a  new  module  mold  in  which 
to  over-mold  the  assembly.  This  provided  us  with  an  opportunity  to  fix  these  oversights.  Accordingly, 
we  increased  the  under-eye  thickness  for  more  lifelike  appearance;  added  a  skin  fold  to  upper  eyelid; 
added  lacrimal  caruncle  for  added  realism;  and  added  decorative  Meibomian  gland  pores  to  the  eyelid 
margins. 

Underlying  the  revised  skin  layer,  the  modules  contain  a  parts  cradle  made  of  Shore  0030  silicone,  a 
tarsus  with  sacrificial  wax  canaliculus  and  mesh  substrate  made  of  Shore  20A  silicone,  a  periorbital  fat 
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layer  made  of  Shore  0010  silicone,  a  conjunctiva  layer  made  of  Shore  10A  silicone  and  an  eye  globe 
made  of  Shore  20A  silicone.  The  tarsus,  canaliculus,  and  periorbital  fat  layers  were  installed  in  the 
cradle  using  RTV  silicone  glue.  These  parts  were  then  over-molded  at  80psi  with  Shore  20A  silicone 
pigmented  to  resemble  skin.  The  resulting  part  was  then  placed  in  an  oven  to  allow  the  wax  part  to  melt 
out.  The  evacuated  canaliculus  was  then  reamed  out  with  a  flexible  plastic  tube  to  ensure  all  wax  had 
been  removed.  The  conjuctival  layer  (Figure  67,  see  also  692. 3. 5. a. iv)  was  then  attached  to  the  back  of 
the  eyelids  with  silicone  glue.  A  circular  cut  was  made  in  the  conjunctiva  which  corresponds  to  the 
outside  edge  of  the  cornea.  The  conjunctiva  was  then  glued  to  the  eye  globe,  allowing  the  globe  to 
move  freely,  while  still  being  firmly  attached  to  the  module.  The  final  steps  would  involve  painting, 
creating  a  laceration,  and  applying  moulage  and  hair. 


Figure  66:  Module  cradle;  cradle  mold;  cradle  cover 

During  the  assembly  and  construction  of  the  first  module,  it  became  apparent  that  replacing  silicone 
RTV  glue  with  2  part  silicone  of  the  same  type  as  the  outer  skin  would  decrease  the  chances  of  failure. 
The  silicone  in  its  liquid  stage  flows  readily,  filling  in  all  the  gaps  between  the  cradle  and  the  tarsus. 


Including  the  conjunctival  layer,  improved  eye  globe,  and  a  tarsus  with  integrated  canaliculus/lacrimal 
duct,  we  produced  three  such  modules. 


Figure  67:  Revised  external  skin  surface  model  with  additional  anatomical  landmarks  and  features;  CAD  model  of 
sacrificial  wax  canaliculus;  assembly  of  soft,  subcutaneous  tissues,  fat  deposits,  tarsal  plate  and  tendons;  partially 
installed  conjunctiva;  overmolded  skin  layer  with  installed  globe. 
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MEETs  Dr.  Freitag,  one  of  our  SMEs  had  mentioned  that  real  skin  is  pliant  and  elastic,  yet  still  firm.  After 
repeatedly  grasping  and  pulling  the  skin  of  the  module,  she  concluded  that  while  this  module's  skin  was 
soft,  it  was  not  elastic  enough.  She  explained  that  human  skin  would  "give"  more  than  the  simulated 
skin  did.  In  order  to  achieve  this  elasticity,  we  determined  that  the  skin  should  be  a  multi-layer 
structure,  consisting  of  a  thin,  firm  outer  layer  sitting  atop  a  very  elastic,  pliant  layer.  This  more  closely 
mimics  the  anatomic  structure  of  skin,  which  is  composed  of  a  tougher  epidermis  and  a  more  pliant 
dermis/subcutaneous  layer. 

A  test  sample  was  constructed,  comprised  of  a  thin  layer  of  Shore  20A  silicone,  followed  by  a  thicker 
layer  of  Shore  0010A  silicone  with  Smooth-On  Slacker  tactile  mutator.  The  test  sample  exhibited  the 
characteristics  that  our  consulting  physician  described.  When  grasping  the  skin  with  forceps  and  pulling, 
the  skin  appears  to  separate  from  the  underlying  mass  and  deforms.  Upon  releasing  the  skin,  it  returns 
to  its  original  shape.  Applying  this  technique  to  all  future  iterations  of  the  modules  adds  only  one 
additional  step. 

We  also  corrected  the  routing  of  the  lacrimal  duct  hose  connector  that  mates  with  the  port  in  the  skull 
that  leads  to  the  nasal  airway.  Instead  of  routing  the  lacrimal  duct  out  the  back  of  the  module,  the  duct 
is  now  routed  internally.  The  duct  terminates  in  the  nasal  passage  of  the  simulator,  which  more  closely 
reflects  the  anatomical  structures. 


Figure  68:  Mold  insert  for  revised  canalicular  routing  in  tarsal  plate,  prior  to  removal  of  flashing;  mold  for 
replication  of  revised  design. 

After  overmolding  several  version  2  tarsal  plates  in  the  cradle,  we  determined  that  the  part  should  be 
decreased  in  thickness.  Both  version  1  and  version  2  produced  an  unacceptably  thick  eyelid  that 
detracted  from  the  user  experience.  Version  3  is  identical  to  version  2  except  that  the  thicknesses  of  the 
superior  and  inferior  tarsus  have  been  decreased  from  .075"  to  .06".  The  results  are  favorable  and  more 
closely  reflect  the  anatomical  structures;  the  design  retains  the  possibility  of  decreasing  the  thickness 
further  should  this  be  recommended  in  the  future. 
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Figure  69:  Tarsal  plate  wax  positive;  corresponding  mold  for  version  3 


Late  in  the  program,  the  design  of  the  canaliculi  was  once  again  refined,  following  a  meeting  with 
representatives  of  FCI  Ophthalmics  (a  potential  licensee  of  the  technology)  to  achieve  a  more  realistic 
match  with  normal  anatomy.  These  soft  tissue  structures  do  not  appear  in  the  CT  imagery  we  used  for 
the  basis  of  our  model,  and  anatomy  texts  generally  do  not  have  complete  sets  of  views  of  the  canaliculi 
and  lacrimal  sac  from  which  3D  anatomy  can  easily  be  determined.  The  image  from  Grey's  anatomy  is 
typical,  leaving  out  the  depth  dimension.  The  CAD  model  used  to  create  the  facial  mold  shows  the 
inward  pointing  section  of  the  canaliculi  that  mates  with  the  lacrimal  sac,  which  remains  with  the  skull. 


Figure  70:  (Left)  Front  view  of  anatomy  from  Grey's  Anatomy.  (Right)  View  from  slightly  above  plane  of  eye  globe, 
showing  canaliculi  merging,  then  turning  inwards  into  the  head  before  meeting  the  lacrimal  sac. 

These  developments  of  anatomical  components  of  the  periorbital  tissue  were  absorbed  into  the  later 
revision  of  the  modules  that  encompasses  the  entire  face,  which  is  described  in  later  sections. 

2.3.5.a.iv.  Eye  globe  tissue  modules 

Given  the  change  from  the  original  static  eye  globes  to  mobile  versions  and  the  simplicity  of  the  original 
eyelid  models,  development  and  improvement  of  the  anatomical  components  of  the  eye  extended  from 
Year  2  through  completion  of  the  project,  and  are  envisioned  to  continue  to  evolve  as  the  system  is 
revised  pursuant  to  future  research  and  commercialization  efforts. 
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The  early  (Year  1)  spherical  globe  models  were  replaced  with  partial  globes  that  include  the  anterior 
segment  and  the  part  of  the  globe  forward  of  the  equator,  and  a  conical  seat  behind  that  mates  with  the 
motion  system  and  fluidic  supplies  for  aqueous  humor  and  blood  simulants  necessary  for  lacerated 
globe  and  retrobulbar  hemorrhage  scenarios. 


Figure  71:  Year  2  status  of  concept  design  for  globe  segment  model  with  initial  conjunctiva;  glove  molded 
conjunctiva  (hollow,  silicone  sphere). 


During  Q6,  we  explored  a  two  part  assembly  design:  one  for  the  eye  lid  and  orbital  rim,  one  for  the 
globe.  This  approach  would  permit  mixing  and  matching  of  different  kinds  of  trauma  modules,  from 
simple  lid  laceration  with  no  globe  involvement,  to  lid  avulsion  and  lacerated  orbit.  The  globe 
component  would  be  compatible  with  the  symptoms  of  retrobulbar  hemorrhage  -  proptosis  and 
hemorrhagic  chemosis.  Shown  below  is  the  bony  orbital  rim  described  above  overlying  a  multi-layer 
flexible  conjunctiva  structure.  The  blue  layer  would  form  part  of  the  globe  module,  bonded  to  the  globe 
at  the  margin  of  the  cornea,  extending  to  the  seat  in  which  the  orbital  rim  is  mounted.  This  would  be 
semi-transparent,  so  simulated  blood  would  be  visible.  The  red  layer  would  be  mounted  to  the  skull  and 
seal  the  fluid  space  between  red  and  blue,  separate  from  the  mechanical  and  electrical  spaces  within 
the  skull.  The  tubes  and  ports  that  connect  to  the  globe  would  provide  an  inlet  for  simulated  blood  to 
enter  this  space.  Removal  of  the  eye  lid  and  globe  portals  would  provide  access  to  the  fluid  space  for 
ease  of  cleaning  and  inspection.  This  concept  was  superceded  by  later  developments,  but  remains  an 
option  for  future  work. 


Figure  72:  Concepts  for  flexible  conjunctiva  layers  and  sealing  surface  to  protect  mechanisms  from  fluid  contact. 
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At  the  end  of  Year  2,  we  developed  a  method  for  fabricating  a  conjunctival  layer  that,  like  the  true 
anatomy,  seals  the  external  environment  of  the  trauma  module  from  the  interior  by  connecting  the 
limbus  of  the  globe  (the  mating  zone  between  the  cornea  and  the  sclera)  to  the  interior  surface  of  the 
tarsus  with  a  continuous  layer  of  thin  silicone  tissue.  The  conjunctiva  layer  is  formed  by  brushing 
silicone  onto  a  spherical  mold  in  multiple  layers,  removing  the  hollow  sphere  from  the  mold  and  the 
inverting  half  of  it  so  that  a  two  layer  structure  that  conforms  to  the  outer  surface  of  the  eye  globe  is 
created.  A  circular  opening  cut  into  the  sphere  aligns  with  the  cornea  and  an  almond-shaped  opening 
opposite  the  circular  cut  corresponds  with  the  opening  of  the  eye  lids  and  provide  alignment  features  to 
support  bonding  of  this  structure  to  the  inside  of  the  eyelid  module  and  the  outside  of  the  globe.  This  is 
described  in  more  detail  below  and  in  the  context  of  the  full-face  modules  described  in  section 
2.3.5.a.vi. 

During  Year  3,  a  new  technique  for  the  creation  of  the  eye  globe  was  developed.  The  previous 
technique  for  making  the  eye  globe  involved  injecting  a  3D  printed  mold  with  silicone.  The  resulting 
globe  had  noticeable  parting  lines  that  detracted  from  the  realism  of  the  simulator  (see  Figure  2). 
Similarly,  the  iris  was  cast  from  a  3D  printed  mold  and  lacked  realism.  Borrowing  from  the  successful 
technique  used  in  creating  the  conjunctiva,  the  new  eye  globe  is  made  brushing  successive  thin  layers 
onto  a  model  of  the  eye  globe.  The  model  consists  of  a  3D  printed  hemispherical  globe,  with  2  recessed 
inserts.  The  first  insert  in  the  shape  of  the  lens  is  inserted  into  the  globe  model.  A  deliberate  gap,  which 
is  later  filled  in  with  silicone,  supports  the  lens.  The  top  of  the  lens  insert  provides  a  surface  to  which 
the  iris  can  be  mounted.  The  iris  is  created  by  brushing  tinted  silicone  onto  the  back  of  the  cornea 
insert,  attaching  it  to  the  lens  insert  and  allowing  it  to  cure.  The  iris  is  the  painted  using  silicone  pigment 
to  resemble  a  human  iris,  and  then  is  again  placed  between  the  iris  insert  and  lens  insert.  A  layer  of 
white  tinted  silicone  is  then  brushed  onto  the  globe  model.  While  that  is  curing,  the  lens  inset  assembly 
is  then  inserted  into  the  globe  model.  After  curing,  a  layer  of  clear  silicone  is  bushed  over  the  entire 
assembly,  capturing  the  iris  in  a  by  a  narrow  band  around  its  circumference.  Once  cured,  the  silicone 
part  is  lifted  off  the  model  and  the  inserts  are  removed  from  the  back,  resulting  in  a  smooth,  hollow 
hemisphere  of  silicone  with  a  precise  air  gap  between  the  cornea  and  iris  and  a  shelf  to  hold  the  lens.  A 
base  is  then  fabricated  using  the  same  technique.  Silicone  is  brushed  onto  the  base  model  and  while  still 
uncured,  the  globe  part  is  placed  on  top  of  it.  The  entire  assembly  cures  as  one  seamless  part.  Excess 
silicone  around  the  base  is  trimmed  as  the  final  step.  The  result  is  a  hollow  silicone  eye  "globe",  with 
two  puncture-able  orifices  for  fluid  access.  A  clear  solution  (like  the  PVA  gel  identified  in  Year  1)  can  be 
injected  through  the  orifice  filling  the  globe.  This  version  of  the  globe  was  replicated  and  presented  to 
cornea  specialists  at  the  2014  USUHS  Ocular  Trauma  Course  for  review  and  testing.  They  found  that  the 
cornea  could  be  lacerated  to  create  realistic  wounds,  and  subsequently  sutured  to  create  a  fluid-tight 
seal,  similar  to  that  of  the  real  cornea  (Figure  73). 
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Figure  73:  Globe  mold  with  inserts  for  lens  and  cornea  spaces;  finished  globe;  view  of  cornea  specialist  suturing 
laceration;  view  of  simulated  cornea  repair  through  (standard,  non-AR)  operating  microscope. 

In  Year  4,  a  new  method  for  creating  the  eye  globe  and  conjunctiva  was  developed.  The  methods 
described  above  were  time  consuming  and  labor  intensive  and  had  a  high  rate  of  variability  in  the 
product.  Rapid  prototyped  molds  were  developed  that  could  cast  a  bi-cameral  globe  and  conjunctiva  in 
a  single  injection.  The  mold  was  made  in  6  parts  and  was  created  first  using  the  Object  260V  3D  printer, 
and  later  by  a  Dimension  Elite  FDM  3D  printer.  Molds  created  using  the  Objet  can  prevent  thin  silicone 
structures  from  curing  fully,  leaving  some  gelatinous  regions,  due  to  the  presence  of  miniscule  amounts 
of  support  material  which  can  cause  cure  inhibition.  No  suitable  solvent  was  found  to  remove  the 
support  material.  Alternative  molds  were  printed  using  the  FDM  style  printer.  These  molds  do  not 
inhibit  silicone.  The  surfaces  were  polished  to  a  high  gloss  in  order  to  achieve  clear,  smooth  castings. 
Especially  careful  design  of  the  FDM  parts  allows  us  to  have  very  tight  control  over  the  structure  of  the 
eye  globes.  The  new  corneas  are  0.5mm  in  thickness  at  the  center  and  gradually  thicken  to  1mm  at  the 
edge.  The  conjunctiva  maintains  a  uniform  0.5mm  thickness  throughout.  The  sclera  has  a  similarly 
controlled  thickness.  The  result  is  an  easily  reproducible,  accurate,  bi-cameral  globe  with  a  conjunctiva 
bonded  at  the  limbus  (Figure  74).  The  lens,  iris,  and  backing  plate  are  fabricated  separately  and  are 
assembled  using  silicone  epoxy.  The  resulting  assembly  can  be  filled  with  two  different  materials— saline 
solution  in  the  corneal  chamber  and  PVA  slime  (or  other  viscus  material)  in  the  scleral  chamber  -  a 
further  improvement  that  matches  real  anatomy. 


72 


Figure  74:  Year  4  eye  globe  mold,  disassembled  (left);  Year  4  eye-globe  and  conjunctiva,  detail  (middle);  eye  globe  and 
conjunctiva  (right) 

2.3.5.a.v.  Neck  motion  system 

At  MMVR  2012  in  the  TATRC  exhibit  area,  RDECOM  had  on  display  a  remotely  controlled  lower  limb 
hemorrhage  simulator  (prototype  of  the  KForce  (Fairfax,  VA)  Trauma  FX  MATT  system),  which  included 
animatronic  elements  that  allowed  the  legs  to  flail,  making  placement  of  tourniquets  realistically 
difficult.  Similarly,  for  the  conscious  patient  in  need  of  airway  management  (intubation,  surgical 
airway),  the  obstructed  airway,  the  gag  reflex  on  intubation  or  the  pain  of  making  the  incisions  for 
cricothyroidotomy  are  likely  to  result  in  conscious  head  motion  and  make  the  process  of  performing  the 
procedure  more  difficult. 

During  Q4,  we  began  the  early  stages  of  adapting  a  neck  motion  mechanism  technique  that  was  partially 
developed  for  our  COMETS  trauma  mannequin,  to  allow  for  both  passive  motion  of  the  head  and  neck 
over  the  normal  range  of  motion,  and  generate  active  motions  to  jerk  the  head  from  side  to  side.  Figure 
75  shows  a  preliminary  design  for  a  McKibben  (air)  muscle  system,  allowing  for  motion  in  pitch,  yaw  and 
roll. 


Figure  75:  design  for  test  platform  for  generating  neck  motions  (pitch,  pan,  roll)  using  McKibben  muscles. 

Early  in  Q5,  we  performed  initial  testing  of  single  elements  of  the  pneumatic  (McKibben)  muscles  that 
would  generate  neck  motion  and  neck  compliance  and  a  solenoid  valve  controller  for  simultaneous 
control  of  all  of  the  muscles.  We  also  contacted  Dr.  Yong-Lae  Park  at  the  Harvard  Wyss  Institute 
(Harvard  Biorobotics  Laboratory)  for  advice  on  optimizing  the  air  muscle  design  and  developed  a 
controller  to  test  the  motion  capabilities  of  the  current  system. 
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Dr.  Park's  advice  led  to  selection  of  an  alternate,  looser  weave  braid  material  that  allows  for  a  larger 
expansion  limit  of  the  muscle  and  thus  a  larger  contraction.  In  addition,  a  closer  fit  between  the  internal 
diameter  of  the  braid  and  the  external  diameter  of  the  inner  pneumatic  tube  prevents  a  buckling 
phenomenon  that  we  observed  in  our  early  testing. 


Figure  76  shows  the  initial  solenoid  valve  assembly,  a  modular  system  that  we  employed  previously  and 
a  basic  user  interface. 


Figure  76:  Neck  motion  test-bed  design,  initial  structure  fabrication,  test  system  with  enhanced  muscle  mesh, 
electrical  control  system,  GUI  for  8-direction  motion  control. 

In  Q6,  we  upgraded  the  pneumatic  valving  system  to  make  use  of  one  3-way  latching  valve  for  each  air 
muscle  (supplies  pressure  or  exhausts  the  air  muscle),  supplied  from  a  common  manifold.  The  air 
muscles  exhaust  into  a  common  second  manifold,  which  in  turn  is  exhausted  to  atmospheric  pressure 
through  a  series  of  relief  valves,  so  that  air  the  muscles  are  exhausted  quickly  or  slowly  depending  on 
which  relief  valve  is  activated. 

As  described  above  in  the  context  of  the  eye  motion  controller,  earlier  tests  made  use  of  an  Arduino 
2009  microcontroller.  With  four  latching  input  valves  and  four  latching  exhaust  valves  in  the  pneumatic 
design,  requiring  two  digital  control  lines  each  for  a  total  of  16  digital  outputs,  we  acquired  an  Arduino 
Mega  microcontroller,  which  has  54  digital  input/output  connections.  The  neck  motion  control  software 
was  modified  to  make  full  use  of  the  Mega's  capabilities,  and  additional  capabilities  added  for 
interfacing  with  the  mannequin. 

With  this  more  capable  system,  we  extended  the  basic  function  tests  performed  earlier  to  perform  the 
same  tests  using  the  mannequin's  power  and  pneumatic  supplies,  crucial  in  creating  the  mannequin- 
integrated  system  that  was  one  of  our  primary  goals. 
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Figure  77:  Pneumatic  control  system  with  fully  enabled  latching  valves  for  four  air  muscles  and  four  exhaust  back¬ 
pressure  valves.  Arduino  Mega  and  control  circuitry  in  prototype  form  on  mannequin's  chest,  valves  and  manifolds 
on  right  shoulder,  pressure  regulator  on  left  shoulder,  test  platform  for  air  muscles  adjacent  to  neck-region. 

We  conducted  investigations  into  optimizing  the  McKibben  muscles  for  low-pressure  supply  air  from  the 
mannequin  (rated  at  approximately  1  atmosphere  by  Laerdal),  by  sourcing  thinner-wall  elastic  tubing  for 
the  muscles'  internal  bladders  and  changing  the  fitting  diameters  to  minimize  pinching  that  distorts  the 
muscle  end  geometry.  Additional  details  are  described  in  section  2. 3. 9. a. 


Figure  78:  Air  muscle  images  from  Q5  showing  pinching  by  hose  clamps  at  muscle  ends.  Alternate  design  under 
development. 
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2.3.5.a.vi.  CT-derived  face,  head  cowl  and  airway  development 
The  preceding  description  of  soft-tissue  component  design  and  fabrication  techniques  was  based 
primarily  on  the  use  of  the  original,  life-cast  version  of  the  head  form,  useful  for  development  and  the 
teaching  opportunity  of  the  USUHS  Ocular  Trauma  Course.  This  limited  anatomy  did  not  support 
development  of  trauma  modules  for  the  right  eye  or  the  remainder  of  the  face  and  the  neck,  necessary 
for  the  facial  trauma  and  cricothyroidotomy  module  installation.  During  the  later  months  of  Year  3,  we 
completed  extraction  of  the  facial  contours  and  soft  tissue  structures  from  the  CT  anatomy  used  to 
create  the  skeletal  features  described  in  this  and  earlier  reports  (not  human  use).  This  included  the 
contours  of  the  full  airway,  including  oral,  nasal,  pharyngeal  and  laryngeal  spaces.  We  created  CAD 
models  of  all  of  these  structures  and  partitioned  them  into  relevant  sections:  a  full-face  module  that 
replaces  the  single-eye  module;  a  cowl  that  includes  the  top  and  back  of  the  head  skin,  ears  and  neck; 
nasal  airway;  oral/pharyngeal/laryngeal  airway.  Certain  soft  tissue  structures,  especially  in  the  neck, 
were  difficult  to  segment  from  the  CT  anatomy,  so  anatomical  references  and  advice  of  experts  in  these 
areas  was  called  on  to  fill  data  gaps.  This  was  also  true  of  the  mandible/maxilla  due  to  extensive  image 
artifacts  in  the  original  CT  scans  that  generated  image  artifacts.  Figure  79  shows  preliminary  models  of 
the  face,  nasal  airway  and  cowl,  while  the  final  version  of  these  components  is  shown  in  Figure  80. 


Figure  79:  Soft  tissue  models  derived  from  CT  rather  than  life-cast  sources.  These  models  will  serve  as  the  basis  for 
all  future  development  of  the  system 

In  Year  4  a  new  skin  was  developed  for  the  CT  derived  skull  described  above  and  neck  in  section  2. 3. 9. a. 
The  skin  takes  the  form  of  a  front  mask  comprising  the  forehead,  eyes,  nose  mouth,  chin,  and  laryngeal 
area,  and  a  cowl,  comprising  the  skin  of  the  back  of  the  head,  ears,  and  neck.  The  cowl  is  resident  on 
the  simulator  and  is  not  designed  to  be  removable.  It  mates  to  the  contours  of  the  back  of  the  skull  and 
is  fixed  rigidly  in  place  to  a  baseplate.  The  face  mask  is  attached  to  the  cowl  via  a  zipper.  The  zipper 
attachment  is  secured  to  the  face  and  head  skin  using  a  silicone  epoxy.  Early  prototypes  were  held  in 
place  with  suture,  however  this  was  labor  intensive  and  has  generally  poorer  results.  Repeated  use  of 
the  zipper  mechanism  at  the  2015  Ocular  Trauma  Course  shows  it  to  be  a  durable  and  reliable  solution. 

The  face  mask  is  designed  to  be  a  consumable  part,  but  it  can  be  reused  a  number  of  times  before  being 
replaced.  Unlike  the  previous  head,  the  new  head  allows  the  use  of  both  eyes.  Additionally,  the  head 
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features  an  insert  for  performing  cricothyroidotomies.  This  insert  can  be  replaced  an  unlimited  number 
of  times  without  adversely  affecting  the  face  mask,  and  because  of  its  smaller  size  and  the  simplicity  of 
its  fabrication,  would  be  relatively  inexpensive. 


Figure  80:  (clockwise)  new  face  and  cowl  installed;  new  face  and  cowl  partially  disassembled;  eye/eyelid  detail; 
cowl  disassembled. 

The  fabrication  technique  used  in  the  new  head  design  has  evolved  on  similar  lines  to  the  new  eye 
globes  described  in  section  2.3.5.a.iv.  The  modular  mold  for  the  face  mask  was  created  using  the 
Dimension  FDM  printer.  This  allowed  for  very  high  pressure  injection  of  parts.  Due  to  the  complexity  of 
the  eyelid  structures  and  the  need  for  repeatable,  high  quality  results,  it  was  decided  that  the  tarsus  and 
canaliculous  should  be  overmolded  onto  a  freshly  cast  face  part.  The  mold  can  be  configured  to  allow 
the  face  (including  the  eyelid  skin)  to  be  cast  in  one  step.  Once  cured,  a  modular  insert  can  be  removed, 
and  a  different  module  inserted.  This  module  contains  the  features  needed  to  cast  the  tarsus.  The  tarsal 
module  also  allows  for  the  insertion  of  a  sacrificial  canalicular  structure  similar  to  the  one  described  in 
section  2.3.5.a.iii.  Nylon  mesh  is  also  installed  prior  to  casting.  Once  installed,  the  tarsus  is  injected  at 
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high  pressure.  The  eye  globes  with  attached  conjunctiva  (described  above)  are  permanently  affixed  to 
the  face.  The  eye  globes  attach  to  the  eye-motion  mechanism  by  three  hose  barbs.  The  result  is  a 
seamless,  perfectly  bonded  tarsus  attached  to  the  back  of  the  eyelid.  This  process  simplifies  and 
dramatically  improves  the  quality  of  the  result  as  compared  to  earlier  techniques.  This  method  greatly 
reduces  the  complexity  of  the  fabrication  process  and  is  similar  to  the  way  in  which  the  parts  would  be 
reproduced  in  industry. 


Figure  81:  Rapid  prototyped  molds  for  face  (left)  and  eye  globe/conjunctiva  (right) 


A  face  was  created  in  Year  4  to  work  in  conjunction  with  the  maxillofacial  fractures  described  above. 
The  simulated  facial  injuries  created  are  intended  to  be  are  too  extensive  to  allow  for  intubation  nasally 
or  orally,  and  are  intended  to  steer  the  user  towards  determining  that  a  cricothroidotomy  is  required 
(Figure  82). 

In  the  final  months  of  the  program,  a  mounting  frame  was  constructed  to  allow  the  same  head/neck 
structure  used  for  the  mannequin-integrated  version  of  the  system  to  be  mounted  to  the  standalone 
simulator.  It  was  demonstrated  at  IMSH  2015  in  the  TATRC  Corral  and  can  be  seen  in  Figure  80  and 
Figure  82. 
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Figure  82:  Severe  facial  trauma  face 

As  we  continued  to  solicit  the  input  of  our  SMEs  during  the  final  stages  of  development,  we  recognized 
the  need  to  provide  a  backing  plate  with  appropriate  eye  attachment  points  so  that  the  SMEs  could 
evaluate  a  sample  facemask  without  the  entire  system  being  present.  To  that  end,  we  designed  a  hard 
plastic  backing  plate  for  the  facemask.  The  backing  plate  allows  a  user  to  perform  a  simulated  eyelid 
repair  on  a  facemask  as  they  would  in  the  simulator.  The  backing  plate  (shown  below)  inspired  Dr. 
Colyer  from  USUHS  to  envision  a  stand-alone  version  of  the  ocular  trauma  simulator  that  consists  only  of 
a  facemask  and  backing  plate  for  use  in  a  classroom  setting.  The  lack  of  instrument  tracking  and  stand 
would  severely  limit  the  functionality  of  the  simulator  but  could  be  a  practical  solution  for  budget 
constrained  schools/training  programs. 


\ 

Figure  83:  Stand-alone  facemask,  assembled  (left);  disassembled  (right). 
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2.3.5. b.  Start  developing  augmented  reality  training  guidance  system  for  enhanced  eye 
trauma  simulator. 

While  not  originally  scheduled  for  commencement  until  a  few  months  into  the  program,  efforts  towards 
creating  AR  components  began  nearly  from  the  beginning  of  the  program. 

2.3.5.b.i.  Hardware  Development 

An  early  development  was  repurposing  one  of  the  Model  800  6DOF  sensors  for  use  in  providing 
position/orientation  data  for  a  virtual  reality  head  mounted  display  (VR  HMD),  as  a  concept  example  to 
consider  how  such  a  system  could  be  employed. 


Figure  84:  commercial  VR  glasses  with  mounting  frame  for  magnetic  position  tracker  and  model  800  sensor  from 
proof-of-principle  simulator  (larger  sensor  appropriate  for  this  use  vs.  attachment  to  surgical  instruments). 

Early  discussions  with  Dr.  Mazzoli  identified  some  of  the  surgical  microscopes  typically  in  use  by  Army 
ophthalmologists  stationed  in  Kandahar,  and  contact  was  made  with  Scan  Optics  Pty  Ltd  (Adelaide, 
Australia).  This  firm  uses  Olympus  optics  that  are  modified  with  additional  optical  and  structural 
features,  a  path  that  we  have  paralleled.  Scan  Optics  was  able  to  provide  us  with  structural  CAD  models 
of  the  microscope  head  that  they  modify,  which  we  are  using  in  our  more  recent  design  processes. 

Early  efforts  to  develop  our  own  binocular  microscope  with  integrated  displays  lead  to  one  of  the 
projects  of  one  of  our  graduate  student  interns,  who  initially  created  a  functional  microscope  using 
binoculars  and  inexpensive,  repurposed  optics  (Figure  85). 

The  second  phase  of  this  work  involved  the  acquisition  of  a  standard,  trinocular  microscope  head 
(Amscope  ZM245NT,  Irvine,  CA),  which  was  partially  disassembled  and  additional  beam  splitters  and 
lenses  were  installed  to  investigate  whether  a  small  LCD  display  such  as  that  on  an  iPod  could  be  used  to 
present  useful  data  through  the  microscope  eyepiece  view  (Figure  85).  At  the  time,  our  understanding 
of  microscope  optics  was  developing,  and  the  display  was  over-magnified,  showing  very  large,  distinct 
pixels  through  the  view  rather  than  an  image  of  useful  resolution. 
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Figure  85:  Evolution  of  augmented  reality  microscope  optics.  Schematic  of  binocular  prism  array;  assembled  low- 
cost  test  platform;  commercial  trinocular  microscope  head;  initial  modification  to  eyepiece  prism  array  to  include 
display,  optics,  beamsplitter. 

Subsequent  to  these  developments,  during  Q4,  detailed  investigation  into  optical  design,  including  use 
of  the  thin-lens  approximation  using  principal  ray  analysis  through  a  compound  lens  arrangement,  lead 
to  the  next  design,  which  used  a  25.5mm  width  LCD  display  borrowed  from  a  separate  project  for 
evaluation  purposes.  We  made  use  of  the  parametric  sketching  tools  available  in  the  Solidworks  (DS 
Solidworks  Corp,  Waltham,  MA)  CAD  package,  which  allows  the  definition  of  geometric  constraints  and 
physical  dimensions.  This  served  as  a  graphical  calculation  tool  that  we  used  to  determine  lens  and 
mirror  location,  given  LCD  size,  desired  output  image  size  and  commercially  available  lens  focal  lengths 
and  diameters  (Figure  86).  Figure  87  shows  a  test-bed  arrangement  of  lenses  and  beam  splitters, 
resulting  in  the  two  images  shown  alongside.  They  show  the  raw  view  through  the  microscope  (a 
Laerdal  mannequin  eye,  showing  reflected  LEDs  from  microscope  ring  light),  and  combined  image  of  eye 
with  an  overlaid  large  blue  iris/black  pupil  image.  This  demonstrated  successful  demagnification  of  the 
LCD  display,  so  that  the  full  resolution  of  the  LCD  is  available  for  information  display. 
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Figure  86:  Thin  lens  approximation  implemented  in  graphical  calculator  using  sketching  and  constraint  tools  of 
Solidworks  CAD  program.  Principle  rays  pass  through  lens  focal  points,  creating  real  and  virtual  images.  LCD 
module  at  left;  concave  lens  straightens  converging  light;  concave  lens  focuses  rays  to  eyepiece  lens  location  at 
right.  Also  shown  is  microscope  objective  lens  (bottom)  and  beam  splitter  (diagonal  at  center)  This  schematic  is 
equivalent  to  the  test  system  shown  below,  with  all  reflecting  elements  removed  to  create  an  axially  aligned 
system  (reflections  do  not  affect  the  refraction  and  image  size  elements  of  the  model. 


Figure  87:  (left)  initial  prototype  of  optics  testbed  based  on  thin  lens  calculation  method,  (center)  Plain  image 
through  microscope,  (right)  Overlaid  image  from  LCD  combined  with  center  image. 

During  year  2  we  further  developed  the  augmented  reality  surgical  microscope  system,  combining  a  high 
brightness  LCD  monitor  and  optics  into  a  commercial  stereo  microscope,  to  fuse  the  image  of  the 
surgical  field  through  the  microscope  with  graphical  overlays  generated  by  the  LCD.  Using  a  more 
complex,  3D  graphical  design  tool  (Figure  88)  that  was  derived  from  the  earlier  version,  we  created  a 
design  for  a  relatively  compact,  complete  AR  microscope  system  (Figure  89).  Further  progress  on  this 
element  of  the  work  is  described  in  sections  2.3.7.C  and  2.3. 12. b. 
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I-  Driginal  a 


^xis2  -  splitter  axis 


Figure  88:  Front,  isometric  and  side  views  of  new  3D  optical  path  calculator. 


Figure  89:  Sequential  development  of  AR  microscope  during  year  2.  COTS  trinocular  microscope  head.  Early 
external  display  test.  Initial  inline  test  of  small  LCD  display.  Integration  of  high  brightness  LCD  during  Q5.  CAD 
models  of  microscope  with  more  compactly  integrated  display  and  optics.  Internal  structure  of  AR  microscope 
awaiting  external  shell  fabrication. 

Early  in  the  program  we  acquired  and  began  testing  the  Microsoft  (Redmond,  WA)  Kinect  system,  to 
examine  its  capabilities  and  applicability  for  tracking  gestures.  As  shown  below,  we  can  recognize  the 
head  form  and  the  user's  hand  in  the  depth  map,  however  the  minimum  distance  range  of  the  system 


83 


(intended  for  body  tracking  for  video  game  interactions)  is  somewhat  larger  than  would  be  desirable  - 
the  Kinect  would  need  to  be  mounted  at  considerable  height  above  the  mannequin. 

In  addition  to  the  Kinect,  we  acquired  a  nano-projector  video  projector  (Qumi  Q5,  Vivitek  Corp.),  with 
high  brightness  and  resolution.  This  system  was  tested  to  investigate  color  schemes  compatible  with 
the  flesh-tones  of  the  mannequin  face  and  the  utility  of  projecting  content  onto  and  around  the  head. 
In  addition,  we  tested  the  Kinect  by  coupling  its  imaging  capabilities  with  virtual  buttons  projected 
around  the  face  successfully. 

A  little-known  property  of  the  Kinect  is  that  it  can  be  used  through  mirrors  -  the  projected  infrared  field 
and  the  field  of  view  of  the  IR  camera  can  be  reflected.  We  took  advantage  of  this  property  by  adding  a 
first  surface  mirror  into  the  structural  frame,  positioned  so  that  the  Kinect  field  and  that  of  the  projector 
overlap  the  face  of  the  head  form  and  the  area  around  it.  This  allowed  us  to  reduce  the  necessary 
height  of  the  system  above  the  face  by  at  least  40%.  As  shown  in  Figure  91,  we  installed  the  web 
camera  to  overlap  the  same  area  so  that  we  could  effectively  capture  surgical  instrument  and  hand 
motions  of  relevance. 

Further  development  of  the  AR  systems  is  described  in  section  2.3.7. 


Figure  90:  (left)  Kinect  representation  of  head  form  with  visual  image,  depth  map  and  rendering,  (right)  test 
pattern  projected  onto  head-form  face  with  addition  of  interaction  zones  for  instruments  (equivalent  to  tray 
location),  graphical  representation  of  bleeding  from  zone  of  left  eye. 


Figure  91:  Fields  of  view  of  projection/imaging  components.  In  blue  are:  (left)  video  projection,  (center)  Kinect 
motion  tracking,  (right)  web-cam.  A  first-surface  mirror  mounted  near  the  top  of  the  system  allows  for  reduced 
overall  height;  minimum  focal  distance  for  Kinect  and  projector  would  otherwise  add  12"  or  more  vertically. 


84 


2.3.5.b.ii.  Content  &  Architecture  Development 

The  content  elements  developed  for  the  Phase  I  simulator  (improved  proof-of-concept  system  based  on 
initial  feedback),  including  user  tracking,  data  collection,  results  and  teaching  content  presentation, 
served  as  the  basis  for  the  broader  development  of  the  content  presented  through  the  AR  components 
(as  well  as  normal  video  displays  and  potentially  tablets  or  other  portable  just-in-time  content 
presentation  platforms).  In  support  of  this  effort,  we  developed  our  content  materials  with 
consideration  of  SCORM  compliance  where  relevant/possible,  and  platform  independence  using 
templates  such  at  HTML5  and  cascading  stylesheets  to  manage  the  details  of  presentation  on  diverse 
display  technologies.  A  typical  rendering  of  an  early  node  in  a  scenario's  content  is  shown  below 
(dynamically  generated  based  on  the  size,  nature  of  browser). 


Figure  92:  Rendered  version  of  scenario  content  using  standard  web  browser. 

Content  prepared  in  a  common,  modular  format  and  the  interactivity  enabled  by  the  multiple  sensor, 
user  interface  and  display  options  available  in  our  system,  enable  a  rich  capability  to  present  materials 
to  users  in  forms  appropriate  to  their  skill  levels  and  training  and  to  their  prior  training  sessions  on  the 
simulator.  The  modular  format  provides  for  standardized  templates  that  instructors  can  use  to  generate 
new  content,  and  tailor  how  it  is  presented  based  on  the  common  interface  to  all  of  the  sensor  and 
display  capabilities  of  the  simulator. 

As  described  earlier  as  an  element  of  the  Phase  I  system  (section  2.3.2. b.vi),  the  surgical  gesture 
detection  system  that  we  developed  is  intended  to  identify,  almost  in  real-time,  the  meaningful 
elements  of  surgical  gestures,  to  detect  whether  a  trainee  is  following  the  expert-defined  sequence  of 
elements  of  a  procedure.  When  event-level  data  is  analyzed,  this  substantially  reduces  the 
computational  burden  of  real-time  measurement  and  evaluation,  as  compared  with  continuous  path 
analysis  at  all  times.  It  enables  identification  of  instances  where  more  detailed  analysis  of  the 
instrument  path(s)  is  necessary,  as  it  is  expected  to  be  for  some  elements  of  a  scenario. 
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To  develop  the  system,  we  identified  a  hierarchical  taxonomy  to  describe  relationships  between  raw  and 
derived  data  (e.g.  position,  velocity),  events  (e.g.  threshold  or  ROI  crossings),  and  in  year  1  began  to 
create  tools  to  perform  the  automated  segmentation.  Early  results  and  progress  in  this  area  were 
described  in  more  detail  in  one  of  our  IMIASH  2012  papers  [Ottensmeyer  et  al.,  2012],  attached  in 
section  7.4  of  the  Appendix,  Attached  Publications. 

During  Q4,  we  collected  performance  data  at  the  USUHS  trials,  providing  baseline  data  to  begin  to 
compare  experts  and  novices.  The  sample  images  in  Figure  93,  from  different  subjects  performing 
different  parts  of  the  scenario,  are  illustrative  of  the  different  phases  of  the  analysis  and  show 
differences  between  experts  and  novices. 

The  first  plots  depict  the  raw,  unprocessed  tool  path  data  for  two  USUHS  participants,  over  a  two 
minute  segment  of  an  eyelid  laceration  repair  simulation.  The  top  pair  of  figures  depicts  data  from  an 
advanced  plastics  specialist,  while  the  bottom  two  figures  are  generated  by  a  first  or  second  year 
resident.  Each  different  trace  color  represents  a  different  tool  used.  The  upper  figure  pair  clearly 
presents  a  more  compact  and  less  dispersed  set  of  movements  in  contrast  to  the  lower  figure  pair.  This 
behavior  appears  to  be  consistent  across  surgical  specialties;  our  earlier  research  in  laparoscopy 
simulation  showed  very  similar  gross  characteristics  [Stylopoulos  et  al,  2004]. 


Figure  93:  Raw  instrument  trajectory  data  over  2  minutes  of  lid  laceration  repair.  (Top)  Advanced  plastics  specialist 
using  0.5mm  Castroviejo  suturing  forceps  (grey)  and  Barraquer  needle  holder  (teal),  with  compact  motion 
envelope.  (Bottom)  Early  year  resident  using  0.2mm  Castroviejo  forceps  (black),  0.5mm  Castroviejo  forceps  (grey) 
and  Vannas  scissors.  Blue  points  above  and  below  "patient's"  left  eye  are  calibration  points  for  registration  of 
physical  model  to  position  tracker. 

Figure  94  shows  automated  ROI  crossing  information  (color  coded  bands  -  green  is  near  eye,  blue  is 
near  instrument  tray)  and  instrument  manipulation  (scissor  actions  -  raw  scissor  motion  data  in  second 
plot)  to  identify  cutting  actions  near  to  the  eye,  combined  with  manual  event  recordings  (black 
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triangles),  representative  of  completion  of  a  series  of  knot  tying,  through  the  removal  of  excess  suture 
material  by  cutting.  This  trace  represents  18  minutes  of  events,  dramatically  reducing  the  amount  of 
data  that  needs  to  be  presented  to  represent  performance,  and  converting  the  raw  data  into  surgically 
meaningful  information. 


o 
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Figure  94:  Upper  plot  -  automatically  and  manually  detected  events:  color  bars  show  entry,  residence  time  and 
departure  from  geometric  Regions  of  Interest;  green  triangles  show  scissor  actions  within  Rols  close  to  eye  lids;  red 
triangle  shows  two  cuts  in  rapid  succession;  for  comparison,  black  triangles  record  manual  entries  of  any  cutting 
action;.  Lower  plot  -  raw  data  for  scissor  closure  action. 

During  Q5  we  made  progress  in  the  gesture-driven  sequencing  of  the  instructional  and  feedback 
materials  presented  to  participants.  Rather  than  requiring  that  a  user  interact  primarily  with  keyboard 
or  mouse  to  advance  the  scenario,  the  various  sensing  elements  provide  the  system  with  state  and 
gesture  information.  For  example,  when  setting  up  the  system,  it  detects  whether  the  proper 
instrument  is  connected,  provides  instructions  for  how  to  do  so  (training  the  user  in  how  to  use  the 
system),  and  advances  automatically  when  that  step  is  complete.  Later  in  the  scenario,  the  system 
updates  the  instructional  display  tracking  the  placement  of  sutures,  for  example,  advancing  when  it 
detects  cutting  corresponding  with  the  trimming  of  sutures  and  replacement  of  the  scissors  in  the 
instrument  tray  (Figure  95). 


Figure  95:  Examples  of  gesture  driven  scenario  content.  Simulator  preparation  instructions  driven  by  correct 
sensor/instrument  attachment.  In-scenario  feedback  on  completion  of  series  of  sutures. 
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2.3.5. b.iii.  Developer  interface  improvements 

In  Q5,  the  developer  interface,  used  for  data  collection  and  export  of  the  data  to  the  analysis  tools 
continued  to  evolve.  Features  added  were  horizontal  scrolling  through  the  full  data  set  and  indications 
of  the  location  of  the  current  view  is  relative  to  the  entire  scenario,  zoom  functions,  display  of  the 
details  of  manually  entered  event  keystrokes,  ability  to  change  the  rate  of  playback,  from  4x  normal  to 
l/64x  normal  speed,  and  selection  tools  to  export  only  desired  subsets  of  the  acquired  data. 

The  display  of  the  virtual  scene,  showing  graphical  models  of  the  instruments  and  the  mannequin  head 
were  decluttered,  showing  only  connected  instruments  rather  than  the  whole  available  suite. 

We  added  management  tools  for  the  AVI  video  files  that  correspond  with  the  collected  instrument  data. 
We  created  a  new  software  library  that  can  access  AVI  streams  in  input  and  output,  choose  the  proper 
compressor  in  order  to  save  the  AVI  stream  in  the  best  format  to  optimize  hard  drive  space 
consumption  and  the  CPU  overhead. 

2.3.5. b.iv.  Quantitative  scoring  and  data  analysis 
In  Q5,  the  list  of  quantitative  overall  performance  metrics  included: 

•  Total  Activity  within  ROI 

•  Total  Duration  within  ROI 

•  Longest  Run  within  ROI 

•  Total  Transitions  between  ROIs 

•  ROI  Containment 

•  Adaptive  ROIs  measures 

as  well  as  the  beginning  of  a  method  for  examining  hand  tremor. 

Additional  details  on  the  AR  system  are  covered  in  the  section  2.3.7.C. 

2.3.6.  “Month  8”  -  Design  of  mandibular/maxillary  trauma  modules,  upper  airway  model  for 
use  of  airway  devices,  new  cricothyroidotomy  portal. 

2. 3. 6. a.  Mandible,  maxilla  and  facial  fracture  modules 
During  Q6,  discussion  with  Dr.  Troulis  concerned  the  types  of  facial  fractures  that  would  be  most 
relevant  in  the  context  of  facial  trauma  and  complications  that  would  necessitate  surgical 
cricothyroidotomy.  She  recommended  that  bilateral  mandibular  fracture  can  allow  the  lower  jaw  to 
collapse,  blocking  the  oral  airway.  We  also  discussed  typical  responses  to  jaw  fracture,  including  wiring 
of  the  lower  to  the  upper  jaw  to  stabilize  the  bones,  which  in  turn  helps  to  reduce  or  stop  ongoing  oral 
hemorrhage. 

During  Year  3,  based  on  these  recommendations  and  statistics  showing  frequency  of  different  fracture 
zones,  we  designed  the  jaw  and  facial  components  of  the  full  skull  of  the  head  form,  including  a  series  of 
disconnectable  components  to  simulate  fractures.  The  components  would  need  to  have  realistic 
behavior  when  palpated  through  the  skin,  and  have  the  capability  of  being  reconfigured  to  present 
alternate  fractures  without  complete  removal  and  replacement  of  the  jaw  or  face. 
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For  the  jaw,  three  levels  of  separation  are  available:  the  fully  healthy  mandible,  when  all  components 
are  linked  together;  fractures  to  the  angle  and  the  body,  which  have  33%  and  20%  frequency  among  all 
jaw  fractures,  and  a  version  in  which  a  bone  segment  has  been  lost  from  the  jaw. 


Figure  96:  Preliminary  segmentation  of  mandible  to  support  alternate  fracture  patterns  within  single  jaw  element. 
Lower  teeth  not  shown  -  to  be  mounted  separately  together  with  oral  soft  tissues. 

Similar  considerations  were  used  to  define  the  facial  fractures:  an  outer  "mask"  was  separated  from  the 
rest  of  the  skull  and  subdivided.  Two  face  bone  masks  were  designed,  one  completely  healthy  and 
another  with  a  zygomatic  complex  fracture  (tripod  fracture  -  40%)  and  with  a  LeFort  I  (15%),  which  is  a 
completely  separation  of  the  palate  from  the  rest  of  the  skull.  These  fractures  as  well  as  the  ones 
realized  for  the  mandible  are  the  most  common. 


Figure  97:  Preliminary  segmentation  of  maxilla  and  face  to  support  alternate  fracture  patterns  within  single  skull 
element.  Upper  teeth  not  shown  -  to  be  mounted  separately  together  with  oral  soft  tissues. 

At  the  end  of  the  year,  the  details  of  the  linking  mechanisms  between  the  bone  segments  were 
completed. 

Substantial  work  on  this  topic  was  undertaken  during  Year  4.  The  designs  for  facial  fracture  components 
that  were  shown  at  the  24  June  2014  IPR  meeting,  with  adjustability  to  customize  fixed  vs.  fractured 
bone  elements,  were  fabricated.  Mandibular,  mid-face  and  zygomatic  fracture  components  are 
available.  These  were  not  designed  for  surgical  repair  scenarios,  but  rather  to  provide  realism  to  facial 
trauma  scenarios  for  first  responders,  so  that  mobile  bone  segments  are  palpable  underneath  the  skin 
surface.  Future  funding  that  has  been  awarded  from  CIMIT,  which  will  commence  after  the  conclusion 
of  this  program,  will  include  further  developments  for  surgical  repair  of  facial  fractures. 


89 


As  described,  the  skull  of  the  system  was  divided  into  a  series  of  bone  segments  that  can  individually  be 
locked  in  place,  or  allowed  to  "float",  representing  facial  fractures.  The  floating  segments  are  held 
loosely  in  place  with  elastic  links  embedded  between  mandible  segments  and  between  maxillary/mid¬ 
face  components  representing  Le  Fort  fractures.  The  variety  of  bone  segments  available  allows  for  a 
wide  range  of  potential  fracture  patterns  to  be  presented  to  the  first  responder,  including  those  that  will 
complicate  airway  management.  Further,  for  injuries  that  may  result  in  complete  loss  of  bone 
segments,  individual  jaw  segments  can  be  removed. 


Figure  98:  (Left)  Model  showing  separable  bone  components.  (Middle)  Locations  of  elastic  links  between  main  skull 
and  Le  Fort  fracture  segments.  Screw  hole  for  locking  zygomatic  fracture  component  is  visible  in  the  grey  segment. 
(Right)  Fracturable  skull  with  nasal  airway  segment  showing  stent  passing  through  lacrimal  sac  and  naso-lacrimal 
duct,  and  screws  holding  Le  Fort  fracture  components  in  place  in  locked  configuration. 

As  the  CT  data  set  available  had  significant  artifact  in  the  mandibular  and  maxillary  regions  due  to  the 
presence  of  metal  fillings,  we  adapted  a  free,  publicly  available  model  of  human  teeth  and  created  our 
own  adapter  structure  to  mount  them  to  the  known  structures.  In  addition,  soft  tissues  were  sculpted, 
based  on  the  existing  tissue  and  bone  contours.  The  sculpted  positive,  for  creation  of  molds  for  these 
components,  is  shown  in  Figure  99,  including  highly  detailed  models  of  the  teeth  and  gums. 
Corresponding  elements  for  the  oral  airway  are  described  in  section  2.3.6.d. 
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Figure  99:  Clay  mold  for  gum  and  inner  oral  cavity,  showing  segments  of  teeth  3D  printed  from  downloaded 
models  and  skull  and  mandible  sections  based  on  our  CT-derived  models 

2.3.6. b.  Jaw  motion  and  head  yaw  motion  mechanisms. 

During  Year  3,  we  investigated  normal  human  jaw  motion  to  ensure  that  for  an  uninjured  jaw,  the 
available  range  of  motion  is  correct,  so  that  intubation  or  similar  procedures  would  be  possible. 
According  to  Posselt's  Diagram  [Koolstra,  2002],  the  movement  of  the  jaw  can  be  described  as  pure 
rotation  for  the  first  10°  and  a  rotation-translation  for  the  next  20°.  To  achieve  this  motion,  a  spherical 
pin  slides  in  a  curved  track  that  allows  the  jaw  to  both  slide  and  pivot.  An  open  frame  captures  a  second 
pin  on  the  jaw  to  limit  the  range  of  motion  to  the  normal  human  range  (Figure  100). 


Figure  100:  Posselt  diagram  showing  the  range  of  motion  of  the  human  jaw;  design  of  motion  restriction  frame; 
initial  mechanical  prototype  of  jaw  structure. 

To  generate  motion  controlled  by  the  system  itself,  a  servo  and  transmission  system  was  designed  and 
constructed.  A  single,  continuous  rotation  servo  drives  a  belt  and  slip  clutch  that  allows  the  jaw  to  be 
opened  and  closed  to  the  mechanical  limits  without  needing  precise  position  control  of  the  servo;  a  limit 
switch  detects  when  the  jaw  is  fully  closed  (Figure  101).  The  elasticity  of  the  facial/oral  silicone 
components  aids  in  providing  restoring  force  to  the  open  jaw  to  return  it  to  a  nearly  closed  state. 
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Figure  101:  Jaw  mechanism  showing  pivot  action  acting  through  parallel  links,  leaving  the  oral  space  and  access  to 
the  pharynx  open;  slider  carriage  which  supports  the  forward/back  motion  of  the  jaw  coupled  with  the  angular 
open/close  action;  clutch  mechanism  to  allow  for  manual  back-driving  of  the  jaw  regardless  of  the  position  set  by 
the  mechanism. 

In  addition  to  jaw  motion  and  the  pneumatic  control  of  the  pitch  and  roll  action  of  the  neck,  the  head 
needs  to  be  able  to  rotate  about  the  neck  axis  (left/right  "no"  action).  A  stepper-motor  and  bevel  gear 
mechanism  was  designed  to  mate  with  a  central  flexible  shaft  that  runs  along  the  length  of  the  neck. 
The  shaft  is  anchored  at  the  base  of  the  neck,  and  the  motor  causes  the  top  of  the  neck  to  rotate  with 
respect  to  the  shaft.  A  pair  of  limit  switches  was  included  to  detect  when  the  head  reaches  the  left  or 
right  limits  of  motion  and  a  potentiometer  mounted  to  the  flexible  shaft  detects  relative  motion 
between  the  shaft  and  the  skull. 


Figure  102:  Head  motion  stepper  motor  and  bevel  gear  linking  motor  shaft  (model  shaft  is  longer  than  real  shaft) 
and  upper  fitting  of  flexible  shaft  (red);  positioning  of  limit  switches  and  potentiometer  relative  to  flexible  shaft 
axis;  partial  cut-away  view  of  jaw  motion  and  head  motion  mechanisms,  also  showing  servos  for  blinking  and 
proptosis  motion. 

Simplified  versions  of  the  main  components  of  these  motion  elements  were  fabricated  and  assembled 
to  test  the  concepts,  verify  whether  the  actuators  selected  were  strong  enough,  and  tune  the  slip-clutch 
stiffness  to  allow  for  motor  controlled  jaw  motion  with  manual  back-drivability  in  the  mechanism  (Figure 
103).  The  final  version  is  shown  below  in  Figure  152  and  was  integrated  into  the  head-neck  form  that  is 
compatible  with  both  the  stand-alone  system  and  mannequin  mounting. 
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Figure  103:  Test  assembly  of  jaw  (omitting  fracture  elements),  head  base  with  air  muscle  mounting  sites  and 
flexible  shaft  seat,  full  mechanism  mounted  for  testing. 

2.3.6.C.  Cricothyroidotomy  sensing  system 

Cricothyroidotomy  is  a  standard  technique  for  establishing  an  airway  when  the  mouth  and  nose  are  too 
heavily  obstructed  or  if  the  upper  airway  is  swollen  to  the  point  of  closing  the  airway.  The  technique 
involves  making  an  incision  through  the  skin  covering  the  larynx,  followed  by  cutting  through  the 
cricothyroid  membrane  and  finally  insertion  of  a  breathing  tube  through  the  openings  in  the  skin. 

Ideally,  the  simulator  should  (a)  respond  appropriately  to  any  intervention  that  would  be  successful  on  a 
real  casualty,  and  (b)  detect  when  a  trainee  makes  an  error  and  provide  the  capability  for  identifying 
how  that  error  was  made  to  enable  corrective  instructions  to  be  given.  To  support  this,  we  prototyped  a 
series  of  incision  detection  and  localization  sensors. 

The  first  series  created  low  resistance  path  in  a  grid  or  serpentine  pattern  embedded  in  a  silicone  matrix. 
The  conductor  links  a  ground-terminal  in  the  sensing  electronic  circuit  and  a  digital  input  terminal  with  a 
pull-up  resistor  enabled  on  our  data  acquisition  system.  During  the  pre-incision  state,  the  connection  to 
ground  through  the  sensor  provides  a  low  voltage  signal  to  the  digital  input  terminal.  When  cut,  the 
digital  input  loses  continuity  to  ground,  and  the  pull-up  resistor  raises  the  input  value  observed  by  the 
digital  input  to  the  high  state.  The  images  show  the  low  resistance  of  the  assembled  system,  consisting 
of  a  sandwich  of  silicone  layers  enclosing  a  serpentine  pattern  of  conductive  silver  ink.  For  small 
deformations,  this  system  worked  well,  however  larger  deformations,  such  as  those  that  would  be 
generated  by  grasping  and  lifting  the  tissue  to  position  it  for  cutting  caused  the  silver  ink  to  fracture, 
breaking  contact  along  the  path  even  after  stretching  is  removed. 
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Figure  104:  measuring  resistance  of  serpentine  conductive  ink  trace  embedded  within  silicone.  Silicone  stretches 
as  normal  skin,  but  conductivity  is  lost  as  ink  trace  is  fractured. 

The  next  iteration  examined  conductive  textile  threads  (typically  cotton  thread  with  metal  filaments 
woven  into  the  twist)  and  very  fine  steel  wool,  extracting  fibers  from  the  wool  to  arrange  within  the 
silicone.  Such  materials  did  not  break  upon  stretching,  however  cutting  through  the  silicone/steel 
wool/fiber  was  very  distinctive,  producing  unrealistic  tactile  feedback. 


Figure  105:  Conductive  thread  and  unwoven  fiber;  fibers  embedded  in  silicone  for  testing. 

We  explored  the  use  of  carbon-black  impregnated  rubber  materials,  including  a  one-part  thermo-setting 
RTV  rubber  and  conductor-coated  sponge  neoprene.  The  conductive  RTV  was  found  to  have  highly 
inconsistent  conductivity  between  the  conductive  rubber  bulk  of  the  material  and  the  electrical  contacts 
of  the  data  acquisition  system.  The  conductor-coated  neoprene  was  embedded  within  silicone  to 
protect  the  conductive  layer  from  external  abrasion  and  inadvertent  contact.  Initially  it  provided  the 
desired  conductive  path,  however  after  a  variable,  but  short  time  period,  resistance  rose  until  the 
material  behaved  as  an  insulator,  which  was  interpreted  by  the  system  as  a  user  cutting  through  the 
conductive  path  and  completing  an  incision.  We  believe  that  the  silicone  setting  process  allows  liquid 
precursors  to  soak  into  the  conductive  layer,  causing  it  to  swell  and  driving  the  conductive  particles 
apart  from  each  other.  Without  a  continuous  path  for  electricity  to  flow,  conductivity  is  lost. 
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Figure  106:Test  sample  for  evaluating  conductive  RTV  silicone  impregnated  with  conductive  particles. 


Figure  107:  Test  sample  for  laying  strips  of  surface-conductive  neoprene  into  silicone  skin  analog. 

During  Q4,  additional  study  of  conductive  polymers,  sheets  and  other  materials  allowed  us  to  identify  a 
conductive  adhesive  tape  from  3M,  which  has  a  random  array  of  conductive  fibers  embedded  in  a 
polymer  tape  with  a  pressure  sensitive  adhesive  on  both  sides.  We  experimented  with  a  series  of 
different  arrangements  of  the  tape  as  substitutes  for  the  earlier  conductive  materials,  and  showed 
promising  results.  The  tape  allows  for  relatively  reliable  contact  to  the  data  acquisition  wiring,  and  is 
much  easier  to  install  into  an  anatomical  assembly  of  simulated  skin  (epidermis,  dermis,  hypodermis) 
and  subcutaneous  fat.  Figure  108  shows  the  functional  layers  (including  a  small  reservoir  of  blood 
simulant  for  effect  when  incising),  a  demonstration  of  the  system  as  applied  to  a  commercial  airway, 
and  a  microprocessor  board  sensing  the  location  of  the  incision  made  in  the  layered  assembly.  The  thin 
tape  parts  easily  when  cut  with  a  scalpel,  so  that  the  silicone  and  other  layers  of  the  structure  generate 
the  majority  of  the  tactile  feedback  experienced  by  the  trainee. 

In  Q5,  we  continued  development  of  the  cricothyroidotomy  primary  incision  detection  sensor.  The 
analog  system  is  similar  in  concept  to  resistive  touch-screen  technology.  Three  conductive  layers  are 
insulated  from  each  other.  The  upper  and  middle  "drive"  layers  can  have  voltage  gradients  applied 
across  them  in  orthogonal  directions  -  one  in  the  x  and  one  in  the  y  direction  -  or  can  be  placed  in  a 
high  impedance  state  so  that  no  current  passes  through  it.  The  lowest,  the  detection  layer,  is  connected 
to  an  analog  input  channel.  When  the  conductive  metal  blade  of  a  scalpel  cuts  through  the  3-layers,  the 
blade  creates  a  path  from  both  drive  layers  to  the  detection  layer.  Rapidly  alternating  which  of  the  drive 
layers  is  active  allows  the  detection  layer  to  feed  an  analog  voltage  that  is  proportional  to  the  distance 
across  the  drive  layer  to  the  microcontroller.  In  this  manner,  a  nearly  continuous  output  of  cut  position 
coordinates  can  be  fed  to  the  controller.  The  drive  circuitry  is  shown  in  Figure  109. 


95 


Figure  108:  (top)  test  layers  of  cricothyroidotomy  portal  concept  -  skin,  sensor  layer,  blood  simulant  reservoir, 
subcutaneous  fat;  partially  assembled  test  block,  with  conductive  tape  strips  aligned  perpendicular  to  correct 
cutting  direction;  assembled  cut  sensor,  (bottom)  Example  placement  of  sensor/cric  block  on  commercial  airway 
model;  microcontroller  testing  shows  illuminated  LEDs  following  cutting  through  sensor  elements. 


Figure  109:  Drive  circuit  and  sensor  body  for  cricothyroidotomy  incision  sensor.  Sensor  in  place  on  neck  "skin"  and 
over-molding  assembly  to  encapsulate  the  sensor. 

A  concept  for  a  simplified  version  that  uses  only  two  conductive  layers  and  employs  commercially 
available  neoprene  rubber  with  conductive  surfaces  (previously  used  in  the  original  yes/no  concept)  was 
explored,  however  high  and  variable  contact  resistance  between  the  thin  surface  layers  and  the  incising 
blade  made  this  solution  unsuitable. 
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In  order  to  establish  the  relationship  between  the  resistance  value  and  distance,  we  performed  the 
calibration  test  as  shown  in  below.  The  resistance  value  has  linear  relationship  with  the  distance 
between  the  electric  wires  at  the  edge  of  the  sensor  and  incision  point.  This  test  was  performed  by 
piercing  the  sensor  at  regular  points  along  the  length  and  width  of  the  sensor.  An  extended  cut  does  not 
display  as  linear  a  response,  however  it  is  sufficient  to  indicate  the  direction  and  length  of  an  incision. 

The  cricothyroidotomy  sensor  was  installed  on  a  Laerdal  mannequin  head  that  was  used  for 
development  purposes,  and  shown  as  part  of  the  demo  presentation  to  TATRC  personnel  at 
MHSRS/ATACCC  2012.  The  sensing  system  was  included  in  an  invention  disclosure  that  was  submitted 
during  year  2.  In  a  reconfigured  shape,  it  is  compatible  with  the  airway  model  that  was  developed  after 
year  2,  described  in  the  following  section. 


Figure  110:  Cricothyroidotomy  primary  incision  sensor  calibration  curves;  sensor  in  place  over  commercial  larynx; 
sensor  integrated  into  replacement  neck  skin  band  over  commercial  mannequin  head/neck  (Laerdal  SimMan 
Essential) 

2.3.6.d.  Airway  model 

As  described  in  section  2. 3. 5. a,  we  segmented  the  airway  features  of  the  CT  data,  with  good  results  for 
the  naso-  through  laryngopharynx  and  nasal  cavities  (Figure  111).  Due  to  the  presence  of  dental  filling- 
related  artifact  in  the  CT  images,  we  scaled  an  imported  model  of  teeth  and  created  sculpted  models  of 
the  soft  tissues  of  the  mouth. 


Using  the  CT-derived  anatomical  modules  as  a  basis,  simplified  versions  of  the  soft  tissues  of  the  nasal 
passages  (reduced  complexity  of  nasal  turbinates  to  ensure  a  large  enough  path  for  nasal  intubation) 
were  fabricated.  Due  to  the  complexity  of  the  nasal  cavities,  they  were  cast  in  left  and  right  halves 
which  were  bonded  together  and  installed  into  the  skull.  The  combined  component  includes  passages 
that  are  aligned  with  and  connect  to  the  naso-lacrimal  duct  and  the  lacrimal  sac  in  the  rigid  structure  of 
the  skull,  so  that  stent-based  repair  of  the  canaliculi  can  be  performed  with  standard  instruments  that 
extend  from  the  puncta  in  the  eyelids  out  through  the  nostrils  (Figure  113). 
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Full  models  of  the  teeth  and  soft  tissues  of  the  mouth  including  tongue,  and  the  upper  airway  including 
larynx  and  upper  trachea  have  been  fabricated  and  are  included  in  the  new  version  of  the  head.  These 
are  compatible  with  nasal  and  oral  intubation  and  cricothyroidotomy  (Figure  112,  Figure  113). 

Figure  112  shows  sculpted  positives  for  creation  of  molds,  including  highly  detailed  models  of  an  airway 
that  includes  the  epiglottis  and  vocal  folds,  for  use  as  landmarks  when  intubation  is  performed  on  the 
system  and  a  detailed  tongue.  The  structures  are  compatible  with  future  implementation  of  addition  of 
internal  bladders  to  generate  tissue  swelling. 


Figure  111:  CT  derived  skin  surfaces  with  substantially  uniform  offset,  reduced  thickness  in  the  facial  regions  close 
to  skull  surface.  "Negative  space"  of  airway  including  nasal,  pharyngeal  and  tracheal  regions.  Detail  of  simplified 
airway  with  continuous  passages  through  left  and  right  nostrils. 
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Figure  112:  Sculpted  details  of  larynx  with  epiglottis,  vocal  folds;  gum  and  inner  soft  tissues;  tongue  and  initial  cast 
rubber  iteration. 


Figure  113:  Demonstration  of  continuous  passage  through  lower  canaliculus,  naso-lacrimal  duct  and  out  through 
nose.  Partially  and  fully  assembled  airway  including  nasal,  oral  and  laryngeal  structures. 

2.3.6.e.  Hemorrhage,  CSF/aqueous  humor  fluid  supply  system 
In  parallel  with  the  physical  modeling,  we  developed  early  schematic  plans  of  the  internal  systems  to 
control  elements  such  as  bleeding,  CSF  leakage  through  ears  or  nose  and  channeling  fluid  to  the  eye 
globes  to  generate  a  slow  flow  of  simulated  aqueous  humor  out  of  lacerations  in  the  cornea.  Early 
iterations  of  the  system  conceived  of  the  inclusion  of  vascular  access  to  carotid,  jugular  or  other  vessels, 
however  the  emphasis  on  developing  components  for  use  in  the  Ocular  Trauma  course  lowered  the 
priority  in  favor  of  other  simulator  components. 
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Figure  114:  Section  of  fluid  schematic  concept  for  control  of  blood  simulant,  (left)  Dual  pump  method  for 
generating  arterial  and  venous  pressures  independently,  (right)  Single  pump  method  with  accumulator  and 
regulator. 

In  Year  2,  a  hemorrhage  control  sensing  system  was  explored.  A  network  of  soft,  collapsible  tubing 
"vessels"  links  external  openings  in  the  relevant  anatomy  and  to  a  common  pressure  sensor  internally. 
Fluid  flowing  unimpeded  out  through  the  network  from  a  common  source  would  register  low  pressures 
at  the  sensor;  measured  pressure  increases  as  hemostasis  is  achieved  across  the  set  of  available 
bleeding  sites  as  net  resistance  to  flow  is  increased.  If  fully  implemented,  the  network  of  simulated 
vasculature  would  span  the  entire  head  and  neck  region.  Instead  of  placing  multiple  pressure  sensors 
about  a  series  of  target  zones  for  intervention  where  a  medic  would  apply  pressure  to  stop  bleeding,  we 
studied  detection  of  the  signatures  of  pressure  changes  as  different  sections  of  a  network  were 
occluded.  Using  a  single,  inline,  pressure  transducer  we  could  detect  (and  ultimately  generate  responses 
to  treatment)  in  an  intervention  independent  manner. 

This  system  would  be  of  use  in  generating  advanced  trauma  scenarios  for  first  responders;  when  linked 
to  a  full-body  mannequin,  there  would  be  no  access  to  the  instrument  tracking  features  of  the 
simulator.  Detection  of  the  appropriate  (and  inappropriate)  application  of  direct  pressure  to  control 
bleeding  would  be  a  measurable  means  of  scoring  performance.  As  with  the  instrument  tracking, 
regions  of  interest  in  terms  of  pressure  thresholds,  timing  and  duration  of  application  of  pressure  and 
correct  site  of  application  can  all  be  defined  and  integrated  into  the  software  architecture  as  "gestures". 

Figure  115  illustrates  typical  results  from  a  test  implementation  of  the  sensor/tubing  network  system.  A 
Deltran  DPT-100  pressure  transducer  (Utah  Medical  Products,  Midvale,  UT),  a  low  cost,  relatively  rugged 
device  for  clinical  use  was  used.  Based  on  a  Wheatstone  bridge,  it  is  capable  of  monitoring  pressure 
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from  -30  mmHg  to  300  mmHg.  For  the  tests,  the  sensor  was  coupled  to  an  industry-standard 
instrument  amplifier  (INA125P,  Texas  Instruments,  Dallas,  TX)  which  provides  a  series  of  reliable  voltage 
sources  and  single  resistor  gain  setting,  the  output  of  which  was  sampled  by  our  Arduino  board  for  data 
collection  and  analysis  (Figure  116).  For  the  range  of  sensor  outputs  measured,  the  gain  resistance  was 
15Q.  In  monitoring  the  pressure  signal  from  the  inline  transducer,  changes  in  pressure  of  the  overall 
system  as  well  as  spikes  that  correspond  with  treatment  were  observed.  The  raw  pressure  signal  (blue) 
from  the  sensor  can  be  analyzed  to  extract  the  signatures  (red)  from  applying  or  releasing  direct 
pressure  on  the  hemorrhaging  vessel.  Combined  the  average  systemic  pressure  and  the  change 
signatures,  detection  of  hemorrhage  control  can  be  implemented. 


Figure  115:  Raw  hemorrhage  back-pressure  measured  by  restricting  flow  through  one  or  more  branches  of  the 
vasculature  (blue).  Derivative  of  pressure  (dP/dt)  for  use  to  flag  when  significant  changes  occur,  e.g.  start  or  end  of 
hemorrhage  control  (red). 


Figure  116:  Test  assembly  for  hemorrhage  control  pressure  sensing.  Clinical  pressure  sensor,  signal  amplifier  chip, 
microcontroller  kit  for  data  acquisition. 

The  hemostasis  sensing  method  developed  in  year  2  serves  as  the  means  for  determining  whether  fluid 
flow  is  manually  controlled.  During  Year  3,  we  developed  a  fluid  control  manifold  system  to  channel 
high  and  low  pressure  blood  simulant  and  low  pressure  clear  fluid,  simulating  aqueous  humor  for  the 
globe  modules  and  cerebrospinal  fluid  to  channels  terminating  in  the  nasal  airway  and  the  ears.  The 
system  borrows  a  design  paradigm  that  we  employed  in  our  earlier  COMETS  trauma  mannequin  system 
[Bardsley  et  al.,  2014],  using  two  plates  to  form  the  manifold,  with  ports  and  fluidic  components  on  one 
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side,  and  the  flow  channels  formed  into  the  other  (Figure  117).  Faster  and  with  the  capability  for  finer 
features  than  the  CNC  aluminum  version  developed  for  COMETS,  we  used  the  Objet  3D  printer  to 
fabricate  these  components,  with  the  added  benefit  that  flow  can  be  visualized  through  plates 
fabricated  in  a  transparent  plastic.  Detailed  design  of  this  element  with  mounting  frame  features  to 
mount  it  within  the  space  in  the  upper  skull  is  shown  in  Figure  118.  The  control  circuitry  for  the  valve 
array  has  been  assembled  in  prototype  form  and  control  software  has  been  expanded  to  accommodate 
the  additional  valves.  An  initial  implementation  assigned  one  digital  output  channel  per  valve  from  the 
Arduino  Mega;  the  final  version  employs  a  series  of  port  expanders  that  use  I2C  commands  from  the 
Arduino  using  only  two  wires  (plus  power  and  ground)  to  the  expander,  which  can  be  mounted  in  the 
same  space  as  the  valve  assembly,  minimizing  bus  wiring  between  the  microcontroller,  which  would 
ideally  be  mounted  within  the  chest  cavity  of  the  mannequin  (together  with  access  to  the  mannequins 
blood  supply  and  a  reservoir  for  clear  fluid),  and  the  valve  array  in  the  head. 


Figure  117:  Three-source  fluid  manifold  and  control  system.  Three  banks  of  miniature  valves  source  high  and  low 
pressure  blood  simulant  and  low  pressure  clear  fluid  from  remote  pressurized  reservoirs  in  the  mannequin  body. 
The  upper  six  valves  provide  control  of  clear  fluid  as  a  CSF/aqueous  humor  simulant  to  each  ear,  both  eyes,  the 
nasal  airway  and  one  site  to  be  determined/reserved.  The  low  pressure  bank  supplies  blood  simulant  to  the  ears, 
nasal  airway,  post-conjunctival  space  for  each  eye  and  multiple  bleeding  sites  in  the  facial  module.  The  front  bank 
of  valves  supplies  high  pressure  (arterial)  flow  to  bleeding  sites  on  the  facial  module. 
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Figure  118:  (Left)  models  of  the  manifold  assembly  in  place  on  the  head,  (middle)  two  part  manifold  with  gasket 
structure,  and  (right)  assembled  valve  array,  awaiting  installation  in  the  head. 

In  addition  to  the  mounting  and  channel  components,  we  explored  directly  3D  printing  the  gasket  using 
TangoPlus  soft  rubber  material.  In  previous  projects,  we  used  silicone  sealant  to  attach  manifold 
sections  together,  which  tended  to  force  silicone  into  the  flow  channels  and  make  disassembly  for  repair 
extremely  difficult.  The  use  of  a  transparent  plastic  (Fullcure  720)  for  the  manifold  parts  also  facilitated 
checking  to  ensure  that  the  parts  were  sufficiently  clamped  together  -  leaks  of  red  blood  simulant  could 
easily  be  seen  during  testing. 

2.3.7.  “Month  10”- 

2. 3. 7. a.  User  testing  of  enhanced  eye  trauma  simulator  at  USUHS  or  Madigan  Army 
Medical  Center. 

We  received  approvals  from  our  IRB  and  ORP  HRPO  for  a  second,  and  later  a  third  data  collection 
exercise  and  participation  in  the  2013  (and  2014)  USUHS  Ocular  Trauma  course.  In  addition,  we 
supported  preparation  of  the  Simulator  Validation  Program  (SVP)  proposal  by  Dr.  Marcus  Colyer  and 
colleagues  at  USUHS,  which  called  for  use  of  this  simulator  as  representative  of  a  non-tissue  alternative 
to  training  on  animal  tissues. 

During  Q8,  as  with  the  first  year's  testing,  we  made  25  lid  laceration  modules  available  for  the  students 
and  faculty  to  use,  all  of  which  were  used  over  the  two-day  wet-lab  sessions  of  the  course.  As  with  Year 
1,  student  availability  was  limited  due  to  the  primary  requirements  for  their  participation  in  the  animal 
and  lecture  sessions.  A  software  error  resulted  in  the  omission  of  residents  PGY  year  from  the  database, 
however  we  verified  that  at  least  ten  PGY2  and  PGY3  residents  used  the  simulator.  The  rest  of  the 
modules  were  used  by  faculty,  of  whom  three  were  oculoplastics  specialists. 

All  participants  filled  out  a  survey  form  which  was  requested  by  Dr.  Marcus  Colyer,  course  director,  to 
provide  feedback  on  participant  impressions  about  the  simulator.  This  survey  presented  Likert  scale- 
type  questions,  asking  for  agreement/disagreement  or  relative  opinions  and,  on  the  following  topics: 

•  User  interface  and  teaching  content 

o  Teaching  content  was  clear  and  understandable 
o  Interface  guidance  was  helpful  in  learning  the  technique 
o  There  was  too  much  detail  in  the  teaching  content 

•  Eyelid  laceration  module 

o  Tarsal  plate  hardness  was  too  hard/too  soft 
o  Eyelid  hardness  was  too  hard/too  soft 
o  Resistance  to  needle  puncture  was  too  much/too  little 
o  Friction  of  passing  the  needle  through  the  tissue  was  too  much/too  little 
o  Friction  of  passing  suture  through  the  tissue 

•  Simulator  stand  and  instruments 

o  Structure  of  the  stand  interfered  with  interaction  with  the  surgical  field 
o  Cables  and  connectors  interfered  with  performing  the  procedure 
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o  Cable  management  bracelets  and  attachments  interfered  with  performing  the 
procedure 

o  Instruments  were  easy  to  take  from  and  return  to  the  rack 
o  Sutures  were  easy  to  access 
o  Lights  were  in  a  suitable  position 
o  Lighting  brightness  was  too  dim/too  bright 

For  each  section,  and  open  ended  comment/suggestion  section  was  included. 

We  performed  preliminary  analysis  on  the  data  compiling  the  results  for  the  scored  questions. 


For  the  interface  and  teaching  content,  the  experts  and  novices  showed  the  same  overall  preferences. 
The  majority  agreed  or  strongly  agreed  that  the  content  was  clear  and  understandable  and  that  it  was 
helpful.  Both  groups  however,  disagreed  that  there  was  too  much  detail  in  the  content.  We  interpreted 
this  to  mean  that  there  was  insufficient  detail  and  that  more  should  be  added.  Among  the  comments, 
were  requests  for  additional  detail  generally,  more  specific  instructions  regarding  details  of  the  knots 
and  suture  techniques  to  be  used  in  the  different  phases  of  the  procedure  as  well  as  acceptable 
alternative  methods,  and  information  on  suture  depth  and  tissues  to  approximate. 


Clear  content? 


■  resident 

■  expert 


Helpful  content? 


■  resident 

■  expert 


Too  much  detail? 


■  resident 

■  expert 


Figure  119:  Simulator  content  responses  between  residents  and  experts 
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For  the  eyelid  laceration  module  itself,  experts  and  novices  tended  to  have  similar  responses  as  well. 
Tarsal  plate  hardness  was  judged  almost  evenly  split  between  about  right  or  too  soft;  eyelid  hardness 
was  about  right  or  too  hard  for  most  respondents;  puncture  resistance  was  about  right  for  the  majority 
of  experts  and  residents;  needle  friction  was  about  right  for  the  majority,  and  suture  friction  was  either 
about  right  or  too  much,  evenly  divided  between  experts  and  residents.  Comments  were  fewer, 
however  there  were  requests  for  additional  and  more  obvious  anatomical  landmarks.  These  would  be 
included  through  changes  in  pigment  added  to  the  later  versions  of  the  modules  and  alterations  in  the 
mold  to  include  features  such  as  the  Meibomian  gland  pore,  which  were  present,  but  not  particularly 
deep. 
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Figure  120:  Lid  laceration  module  physical  characteristics  responses 


For  the  simulator  stand  and  instruments,  responses  were  more  broadly  distributed  in  certain  cases.  The 
median  response  was  that  the  stand  did  not  interfere,  however  there  were  some  who  felt  that  it 
interfered  somewhat  (possible  that  the  agree/disagree  statement  was  misunderstood  and  inverted). 
Cables  and  connectors  clearly  were  troubling  for  some,  but  not  for  others.  The  cable  management 
bracelets  were  roughly  evenly  distributed  between  those  who  were  neutral,  disagreed  or  strongly 
disagreed  that  they  interfered  with  the  task.  The  instrument  access  using  the  tray  and  the  connection 
mechanism  were  found  to  be  easy  to  use  by  nearly  all  participants.  The  lighting  system  was  found  to  be 
about  right  by  most,  although  too  dim  for  some  users.  The  majority  of  the  comments  concerned  the 
cabling  attached  to  the  instruments,  with  some  requests  for  wireless  instruments  (which  is  not  possible 
given  current  magnetic  tracker  state  of  the  art  technology  and  requirements  on  the  size  and  mass  of  the 
component  that  would  be  attached  to  the  instruments),  others  suggesting  alternate  cable  management 
approaches. 

Discussions  with  Dr.  Colyer  following  the  course  lead  to  an  invitation  to  participate  in  the  2014  course. 
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Figure  121:  Standalone  simulator  characteristics  responses. 

2.3.7.b.  Pass  upgraded  proof-of-concept  simulator  specifications  onto  commercial 
partner. 

All  of  the  developments  regarding  contact  with  potential  commercial  partners  over  the  entire  program 
were  described  in  section  2.3.3. b. 
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2.3.7.C.  Continue  working  on  augmented  reality  guidance  system  for  enhanced  eye 
trauma  simulator. 

2.3.7.c.i.  Augmented  Reality  Microscope 

In  year  2,  the  AR  microscope  reached  a  functionally  complete  state,  with  the  optics  for  both  left  and 
right  eye  implemented,  measurement  of  the  zoom  level  made  available  by  linking  a  precision 
potentiometer  to  the  microscope  zoom  shaft,  creation  of  power  circuitry  for  the  LCD,  backlight  and 
surgical  light  source,  creation  of  software  tools  to  display  stereoscopic  views  and  mounting  of  an 
Ascension  position  tracking  sensor  to  the  microscope  to  track  its  location  relative  to  the  mannequin 
head.  The  software  architecture  was  modified  to  display  a  model  of  the  microscope  within  the  virtual 
environment  of  the  data  acquisition  interface,  and  the  same  position  tracking  information  is  used  to 
define  the  stereoscopic  imagery  shown  to  the  user  through  the  microscope.  Test  patterns  were 
displayable  through  the  microscope.  Software  tools  were  used  to  align  the  graphical  overlays  of  left  and 
right  eyes  with  each  other  and  with  the  real  object,  and  improvements  to  the  hardware  were  made  to 
rotate  the  horizontally  aligned  images  from  the  LCD  to  the  angled  optical  path  through  the  microscope 
(divergence  of  10  degrees  between  eyes  requires  5  degree  rotation  of  LCD  image  either  in  software  or 
through  mirror  positioning  -  software  rotation  results  in  poorer  image  quality  using  current  LCD). 


Figure  122:  Year  2  AR  microscope  with  enclosure,  internal  LCD,  ring  light  (at  bottom),  position  sensor  mounted  to 
right  of  lower  cowling.  Test  patterns  generated  by  software.  Monocular  view  of  test  pattern  fused  with  optical 
image  of  test  object.  Representation  of  the  virtual  test  object  in  relation  to  a  real  test  object. 

Following  the  earlier  work,  the  software  architecture  was  modified  to  display  a  model  of  the  microscope 
within  the  virtual  environment  of  the  data  acquisition  interface,  and  the  same  position  tracking 
information  is  used  to  define  the  stereoscopic  imagery  shown  to  the  user  through  the  microscope.  As 
the  microscope  is  translated  or  rotated,  the  overlaid  view  changes  to  match.  Test  patterns  were 
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displayed;  the  next  steps  in  this  area  include  implementing  views  of  the  virtual  environment  for  overlay 
onto  the  view  of  the  real  objects. 


Figure  123:  AR  microscope  with  enclosure,  internal  LCD,  ring  light  (at  bottom),  position  sensor  mounted  to  right  of 
lower  cowling.  Graphical  model  of  microscope  in  virtual  environment  for  calculation  of  relevant  display  to  be 
presented  on  LCD  and  viewed  through  eyepieces. 


It  can  be  seen  in  Figure  122  that  the  graphical  images  are  rotated  with  respect  to  each  other  (note 
colored  rings  around  target  object).  This  was  necessary  because  the  optical  paths  of  the  eye  pieces 
diverge  from  each  other  by  10  degrees.  To  make  the  LCD  image  appear  aligned  through  the  eyepieces, 
the  LCD  image  needed  to  be  rotated  by  5  degrees  each  in  the  opposite  directions.  Control  panel 
elements  were  added  to  our  primary  data  acquisition  interface  to  adjust  this  rotation  as  well  as  offset, 
skew  and  scale  the  output  images. 


Figure  124:  Control  panel  for  microscope  view  positioning  and  alignment. 

We  observed  that  while  this  adjustment  aligns  the  images,  the  pixels  grids  viewed  through  the  eyepieces 
remained  at  angles  with  respect  to  each  other.  Image  quality  would  be  improved  further  if  the  pixel 
grids  were  aligned,  which  required  modification  of  the  optical  path  set  up  between  display  and  eye 
piece  lenses. 
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The  earlier  2D  graphical  tool  described  above  assumed  that  the  path  from  display,  through  lenses  and 
mirrors  to  the  eyepiece  lay  on  a  plane.  Relaxing  that  assumption  and  allowing  a  3D  path  permited  the 
generation  of  a  path  with  chirality  (rotation).  The  3D  calculation  tool  set  the  LCD  images  to  be  aligned 
with  the  pixel  array  and  the  output  optical  axis  aligned  with  the  eyepiece.  Four  points  defined  on  the 
LCD  plane  were  projected  through  the  series  of  lenses  and  mirrors  to  their  image  positions.  The  rotation 
of  lines  connecting  pairs  of  image  points  shows  the  rotation  of  the  image.  By  adjusting  mirror  locations 
and  angles,  the  LCD  image  could  be  rotated  optically  (rather  than  graphically),  resulting  in  the  desired 
alignment.  Once  the  mirror  angles  were  calculated,  the  mirror  holding  components  in  the  microscope 
were  modified  to  match,  3D  printed  and  installed,  replacing  the  earlier  versions. 


)Axis2  -  splitter  axis 


Figure  125:  Front,  isometric  and  side  views  of  new  3D  optical  path  calculator. 

During  Year  3,  enhancements  and  improvements  to  the  AR  microscope  and  the  augmented  reality 
display  systems  of  the  simulator  continued. 

As  described  in  further  detail  in  section  2.3.8,  we  added  a  modular  touch-screen  display,  in  addition  to 
the  microscope  display  and  that  of  the  microprojector  (used  to  project  content,  dynamic  graphical 
alterations  to  the  face  of  the  mannequin).  This  required  extending  the  3D  engine  software  to  support 
multiple  rendering  ports,  reducing  the  overhead  required  to  manage  the  microscope  display. 

2.3.7.c.ii.  AV  Recording  and  Compression 

After  the  implementation  of  the  different  components  required  for  the  video  acquisition,  in  Q5  all  these 
software  components  were  combined  in  order  to  create  the  architecture  to  manage  the  video 
acquisition  (webcam  interface,  with  a  manual  decoding  YUV  filter  implementation)  a  codec  manager  (for 
video  compression  into  the  desired  video  format)  and  the  AVI  manager  (already  implemented  in  the 
previous  months. 
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Figure  126:  Updated  data  acquisition  interface  including  new  video  capture  elements.  Available  CODEC  list  and 
camera  settings  at  left.  Instrument  tracking  data  streams  shown  in  colored  bars  at  bottom 

In  year  2,  we  observed  that  image  artifacts  (patches  of  saturated  image)  appeared  in  the  video  captured 
from  user  testing.  While  this  did  not  prevent  use  of  the  video  for  development  purposes,  it  was 
distracting  for  anyone  reviewing  the  video.  In  Year  3  (Q9),  the  YUV  filter  implementation  was  improved 
to  eliminate  the  artifacts. 

Synchronization  between  the  instrument  tracking  data  and  the  video  so  that  the  synthetic  view  playback 
corresponded  with  the  video  playback  was  completed,  enabling  scrolling  through  recorded  data  from  a 
scenario  and  selection  of  sub-elements  for  more  detailed  review  and  analysis. 

2.3.7.c.iii.  Hand/Instrument  Motion  Tracking 

Our  Year  1  tests  of  the  Kinect  system,  suggested  that  the  tracking  resolution  was  insufficient,  in  part 
because  the  minimum  working  distance  between  the  Kinect  and  the  surgical  field  was  too  large.  A  novel 
hand/finger  tracking  device  called  Leap  Motion  (Leap  Motion,  Inc.,  San  Francisco,  CA)  had  recently 
appeared  on  the  market,  and  early  release  of  development  systems  were  made  available  to  developers 
with  interesting  applications,  as  determined  by  review  of  abstracts  submitted  to  the  company.  In  Q6, 
we  obtained  one  Leap  system  and  began  to  evaluate  its  capabilities.  It  appeared  based  on  product 
literature,  that  it  could  detect  up  to  10  finger  tips  within  its  working  space,  a  space  approximately  24"  on 
a  side,  and  that  it  was  much  better  suited  for  hand/finger  tracking  than  the  Kinect. 


Ill 


Figure  127:  Photo  of  Leap  unit,  with  mini-USB  connector.  Product  photo  from  Leap  web  site;  demo  application 
available  with  device  shows  only  line  segments  for  finger  tips,  not  rendered  hands. 

In  Q7,  we  studied  the  possible  uses  of  the  Leap  Motion  finger/hand  tracking  system,  through 
modification  of  the  example  programs  included  with  the  SDK,  such  as  the  Sample_VS2008  project,  which 
demonstrated  access  to  the  raw  data  coming  from  the  sensor.  The  device  performed  similarly  to  videos 
displayed  on  the  vendor  website,  as  which  we  confirmed  for  ourselves  through  use  of  the  example 
programs  and  software  demos  that  it  is  possible  to  download  from  Internet.  The  sensor  provided 
multiple  finger  positions  and  orientations  (each  tracked  finger  in  five  degrees  of  freedom)  and  the  code 
generated  extra  data  (timestamp,  number  of  fingers,  frame  ID).  Mounted  under  a  monitor  positioned  to 
the  side  of  the  simulation  platform,  the  Leap  could  track  surgical  instruments  (possibly  out  of  range  of 
the  tracking  system)  or  surgeon's  gloved  fingers  and  provide  indications  on  the  screen  -  sharp 
instruments  and  gloved  fingers  being  incompatible  with  various  touch  screen  technologies. 

While  initial  experience  showed  that  it  performed  similarly  to  the  vendor's  product  videos,  in  Year  3  we 
revisited  the  use  of  the  Kinect,  which  had  since  been  redesigned  to  include  a  near-field  mode.  We 
studied  the  Microsoft  Kinect  SDK  in  order  to  replace  the  older  and  less  powerful  framework  developed 
from  third  parties  to  integrate  the  Microsoft  Kinect  functionalities  into  the  system.  The  new  version  was 
found  to  be  fast  and  very  stable,  allowing  more  control  of  the  device.  Using  the  Microsoft  Kinect  SDK, 
we  tested  the  Kinect  for  PC  and  the  Kinect  for  X-Box  versions  in  order  to  choose  the  best  one  for  the 
project.  In  addition,  on  both  Kinects,  an  optional  optical  element  intended  to  shorten  the  working 
distance  of  the  Kinect  by  40%  so  that  users  with  small  rooms  can  still  interact  with  the  Kinect  (Nyko 
Zoom,  Nyko  Technologies,  Los  Angeles,  CA)  was  mounted,  but  without  success  -  the  optical  distortions 
introduced  were  found  to  be  unacceptable.  Our  final  design  iteration  for  the  Kinect  is  to  use  the  Kinect 
for  PC  with  Near  Mode  activated. 

Also  in  Year  3  we  evaluated  the  use  of  the  Natural  Point  TrackIR  5,  an  infrared  optical  tracking  system 
developed  for  the  gaming  market,  but  used  extensively  for  gaze  tracking  and  device  tracking  work.  Our 
findings  were  that  the  system's  sensor  noise  was  unacceptable  for  instrument  tracking,  and  the  need  to 
use  a  set  of  retroreflective  markers  or  IR  emitting  LEDs  in  a  pre-determined  configuration  make  the 
system  unsuitable  for  our  current  uses.  In  the  future,  tracking  of  head  motion  as  a  proxy  for  gaze  or 
area  of  attention  may  be  useful,  however  this  is  likely  to  be  outside  the  current  scope  of  work. 
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2.3.7.c.iv.  Event  Driven  Gesture  Analysis 
During  Q6,  we  met  with  several  of  our  subject  matter  experts  to 
continue  our  framing  of  the  educational  goals  of  the  upper  eyelid 
laceration  repair  module.  While  very  difficult  to  depict  graphically, 
our  state  diagram  of  the  entire  procedure  grew  extensively.  Based 
on  input  from  the  SMEs,  we  mapped  the  entire  procedure  from  start 
to  finish,  including  many  actions  beyond  the  scope  of  the  system. 
The  purpose  of  this  exercise  was  to  understand  what  context  the 
user  was  being  placed  in  when  using  the  simulator.  States  of  the 
procedure  such  as  requesting  an  x-ray  of  the  patient  would  be 
carried  out  in  software  as  a  series  of  interactive  steps  in  the  GUI 
rather  than  an  interaction  with  the  simulator  mannequin. 

Several  areas  of  the  training  module,  when  mapped,  started  to 
illustrate  repetitive  functions.  We  captured  these  repetitive 
functions  (actions,  gestures,  thresholds,  etc.)  and  used  them  to  drive 
our  software  development  processes.  For  example,  our  ability  to 
detect  a  specific  sequence  of  events  associated  with  placing  the 
needle  through  the  eyelid  and  retrieving  it  through  the  other  side 
formed  a  basic  building  block  in  our  suturing  gesture  recognition 
system.  If  the  system  detected  a  second  pattern  of  this  type,  we 
would  know  that  the  user  had  taken  two  bites  to  pass  the  suture 
through  the  location  in  the  wound.  Should  these  events  be  followed 
by  a  knot  tying  gesture,  we  would  then  be  able  to  warn  the  user  that 
it  was  advisable  to  leave  the  sutures  untied  until  all  of  the  sutures 
have  been  placed  and  the  lid  margin  approximated.  This  capability  is 
necessary  both  for  real-time,  in-context  surgical  guidance  for  novices 
as  well  as  forming  the  foundation  of  our  gesture  recognition 
framework. 

As  of  Q6,  the  description  hierarchy  tree  had  10  levels  of  description: 

•  Curriculum:  1  entry  (eye  trauma  course) 

•  Module:  1  entry  (marginal  lid  laceration) 

•  Styles:  1  entry  ("textbook"  sequence  -  later  entries  to 
include  surgeon-specific  techniques) 

•  Steps:  5  entries  for  current  style  (e.g.  examination,  suturing 
tarsal  plate,  incision  closure)  -  different  numbers  of  Steps 
will  apply  to  alternate  Styles 

•  Tasks:  52  defined  across  all  Steps,  with  varying  numbers  for 
each  Step  -  marginal  suture  through  tarsal  plate 

•  Gestures:  may  be  simple  (review  X-ray)  or  compound,  with 


►  Training  Modules 
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further  divisions  (e.g.  2-1-1  suture  gesture) 

•  SubO-Gesture:  optional  -structure  for  capturing  needle  passages  without  knot  tying 

•  Subl-Gesture:  high  level  components  of  gestures,  e.g.  single  bite  suture,  2-throw  knot,  1-throw 
knot,  cut  suture 

•  Sub2-Gesture:  mid  level  gesture  component,  e.g.  tissue  grasping,  needle  bites,  repetitions  of 
throws,  suture  grasping,  knot  tensioning 

•  Event:  sensor-driven  events  that  comprise  the  Sub2-Gestures:  instrument  jaw  opening/closing, 
applied  force  level  threshold  transitions,  passage  across  region  of  interest  boundaries,  focused 
path  analysis 

Sample  elements  of  the  hierarchy  are  depicted  in  the  sidebar  above. 

The  relatively  simple  early  approach  we  developed  to  drive  progression  through  the  scenario  content 
(e.g.  instrument  attachment/detachment,  Rol  threshold  crossing)  was  expanded  in  year  2  with  a  richer 
set  of  algorithms  in  order  to  recognize  the  different  gestures  performed  by  the  surgeon  during  the 
procedure.  This  was  based  on  a  more  extensive  set  of  elemental  states,  listed  in  Table  1.  This  list  was 
extended  with  an  initial  set  of  error  conditions  for  triggering  warnings  from  audio  cues  to  video 
instructions  on  correct  technique. 


Table  1:  Elemental  states  detected  by  gesture  algorithm  in  year  2. 


ID 

Routine  Description 

-2 

Stop  state  machine 

-1 

restart  state  machine 

0 

NULL 

1 

Scissor  is  cutting  inside  a  specified  ROI 

2 

true  if  a  forceps  is  connected 

3 

true  if  no  forceps  are  connected 

4 

True  if  a  needle  holder  is  connected 

5 

True  if  no  needle  older  are  connected 

6 

True  if  an  instrument  is  inside  a  specific  ROI 

7 

True  if  a  scissor  enter  inside  a  specific  ROI,  cut  and  after  goes  out 

8 

Record  data  for  scoring 

9 

Plot  scoring  results 

10 

True  if  the  spear  is  connected 

11 

True  if  the  spear  enter  in  a  specific  ROI  and  remains  there  for  at  least  3 
seconds 

12 

True  if  the  forceps  enters  into  a  specific  ROI  and  remains  there  for  at  least  3 
seconds 

13 

True  if  a  double  pass  suture  is  performed  (first  pass) 

14 

True  if  a  double  pass  suture  is  performed  (second  Pass) 

15 

True  if  2-1-1  is  performed  (not  complete) 

ID 


WARNING  CHECKING 
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100 

Check  for  too  hard  forceps  grasping  (A) 

101 

Check  for  too  hard  forceps  grasping  (B) 

130 

Check  for  single  pass  suture  instead  of  a  double  one  (A) 

131 

Check  for  single  pass  suture  instead  of  a  double  one  (B) 

We  investigated  extending  the  scripting  language  that  was  based  on  Boolean  values  by  including  a 
numerical  parameter  for  certain  routines,  in  order  to  create  different  settings  (e.g.  trim  length  of 
sutures)  for  reuse  of  the  same  algorithm  for  different  types  of  users. 

A  series  of  schematics  of  the  state  machine  structure  implemented  in  code  is  shown  below,  with  the 
various  links  between  states  defined  by  the  test  functions  that  determine  which  threshold  has  been 
crossed. 


Figure  128:  schematics  of  elements  of  state  connections  for  different  levels  of  gesture  recognition  hierarchy 

In  addition,  the  XML-based  scripting  language  used  to  manage  simulation  content  was  extended  to 
allow  control  of  graphical  user  interface  elements  (buttons,  sliders)  and  addition  of  comments  to 
improve  readability.  The  entire  gesture  recognition  software  core  was  exported  to  an  external  DLL  to 
permit  reuse  of  the  code  for  future/alternative  surgical  simulators.  It  included  elements  to  manage 
simulation  content  slides  and  the  event  handler  signaling  transitions  between  content  elements,  both  to 
the  correct  next  step  in  a  procedure  and  to  warning/error  states  that  trigger  presentation  of  additional 
teaching  materials.  The  architecture  was  modified  to  split  content  management  and  gesture 
recognition/segmentation  into  two  independent  processes.  For  the  2013  USUHS  session,  a  subset  of  the 
full  content  was  enabled,  with  tools  to  allow  the  instructor  to  drive  content  display/progression. 
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Throughout  Year  3,  the  gesture  recognition  system,  including  algorithms  and  the  (scenario)  developer 
interface  continued  to  evolve  and  improve. 

The  earliest  new  element  was  implementation  of  the  ability  to  perform  the  analysis  process  on  pre¬ 
recorded  data,  rather  than  only  on  live  streams  of  incoming  position/sensor  data.  The  timeline  display 
of  data  was  expanded  to  include  additional  lines  identifying  various  classes  of  events,  with  flags  entered 
where  the  relevant  events  are  detected. 
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Figure  129:  Q9  (Year  3)  user  interface  and  detail  of  event  timeline. 


On  the  timeline  the  events  were  separated  in  order  to  facilitate  the  analysis  and  comparison  between 
automatic  and  manual  gesture  entries.  The  different  tracks  present  on  the  timeline  are: 


•  Tasks  -  representing  the  subdivision  of  the  entire  procedure  in  different  sections  (Cleaning, 
Exploration,  Suturing).  Each  section  is  segmented  based  on  the  findings  of  the  algorithm,  based 
on  which  surgical  instruments  are  connected  and  the  movements  of  those  instruments  across 
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the  different  ROIs.  The  instruments  are  considered  by  category  (e.g.  alternate  forceps  have  the 
same  category,  different  scissors) 

•  LLEv  -  (Low  Level  Events)  identifying  all  the  elementary  events  such  as  instruments  connections 
and  disconnections,  opening,  closing.  This  category  of  events  forms  the  bricks  used  to  create 
higher  levels  of  events. 

•  ROIEv  -  (ROI  Events)  representing  all  events  regarding  the  transitions  between  different  ROIs. 
This  category  of  events  is  still  considered  among  low  level  events  but  is  on  a  separate  track 
because  they  occur  very  often  and  can  create  problems  in  the  analysis  of  the  other  LLEv. 

•  HLlEv  -  High  Level  Events  of  1st  class  are  all  those  events  that  can  be  detected  as  combinations 
of  LLEv  and  maybe  integrating  other  information  available  from  the  surgical  instruments.  This 
category  of  events  is  also  logically  connected  with  the  Task  track. 

All  events  visualized  on  the  timeline  have  different  colors;  for  certain  type  of  events  that  usually  require 
a  beginning  and  ending  of  an  action,  the  labels  appear  at  different  heights  in  order  to  facilitate 
visualization  of  the  beginning  and  ending  of  those  actions. 

Under  this  structure,  the  decomposition  for  the  eyelid  laceration  procedure  is  shown  in  Table  2,  forming 
a  subset  of  the  10-level  hierarchy  described  above,  with  every  level  of  the  hierarchy  is  created  as 
combination  of  events  of  the  previous  levels.  Each  entry  of  the  table  was  implemented  as  single 
algorithm  and  each  algorithm  determines  the  setting  of  the  appropriate  state  variable  or  adds  a  new 
event  into  the  events  list.  Warning  management  was  implemented  as  single  level  structure  where  the 
unique  layer  was  made  up  of  a  set  of  LLE  and  each  event  was  associated  with  a  specific  recovery  action 
or  set  of  actions.  Every  section  (task)  was  subdivided,  having  a  constant  structure  with  a  beginning, 
execution  and  end. 


Table  2:  Year  3  Task  decomposition  for  eyelid  laceration  repair  scenario 


Eyelid  Laceration  Repairing  Gesture  Engine 

Procedure  Analysis  Engine 

Warning  Analysis  Engine 

Proc 

Tasks 

SubTasks 

High  Level  Events 

Low  Level  Events 

Warning  Events 

Eyelid  laceration  repairing  Procedure 

Cleaning 

Begin  Cleaning 

Instrument  Set  Connected 

Spear  Connection 

Improper  instruments  disconnection 

Message  Audio /Video 

Cleaning 

Spear  moving  inside  the  Eye  ROI 

Spear  Tray  ROI  Out 

Timeout  to  begin  the  cleaning 

Message  Audio/Video/Content 

Spear  Eye  ROI  In 

timeout  to  end  the  cleaning 

Message  Audio/Vtdeo 

End  Cleaning 

Spear  back  In  the  tray 

Spear  Tray  ROI  IN 

Proper  Instrument  set 

Message  Audio/Video/Content 

Spear  Disconnection 

Spear  Disconnection 

Exploration 

Begin  Exploration 

Instrument  Set  Connected 

Connect  K1 

Improper  Instrument  disconnection 

Message  Audio /Video 

Connect  FC2 

Proper  Instrument  set 

Message  Audio/VMeo 

FC1  OR  FC2  Out  ol  the  Tray  ROI 

FC1  Tray  ROI  OUT 

Timeout  to  begin  the  Exploration 

Message  Audio/Video/Content 

FC2  Tray  ROI  OUT 

Timeout  to  end  the  Exploration 

Message  Audio/Video 

Exploration 

FC1  working  in  Eye  ROI 

FC1  Eye  ROI  IN 

Too  hard  grasping 

Message  Audio/Video/Content 

FC1  Grasping 

Grasping  out  of  the  proper  workspace 

Message  Audio/Video/Content 

FC1  Releasing 

FC2  working  in  Eye  ROI 

Fa  Eye  ROI  IN 

Fa  Graspmf 

Fa  Releasing 

End  Exploration 

FC2  back  into  the  Tray 

Fa  Tray  ROI  IN 

FC1  Back  into  the  tray 

FC1  Tray  ROI  IN 

Suturing 

Beginning  suturing 

Instrument  Set  Connected 

FC  Connection 

Improper  Instrument  disconnection 

Message  Audto/Video 

NH  Connection 

Proper  Instrument  set 

Message  Audio/Video 

SC  Connection 

Timeout  to  begin  the  Suturing 

Message  Audio/Video/Content 

FCor  NH  in  Eye  ROI 

FC  Tray  ROI  OUT 

Timeout  to  end  the  Suturing 

Message  Audio/Video 

NH  Tray  ROI  OUT 

Too  hard  grasping 

Message  Audio/Video/Content 

SC  Tray  ROI  Out 

Grasping  out  of  the  proper  workspace 

Message  Audio/Video/Content 

Plating  St  it  dies 

Inserting  needle 

NH  Loaded 

Cutting  out  of  the  proper  workspace 

Message  Audio/Video/Content 

FC  Grasping 

dangerous  needle  pulling 

Message  Audio/Video/Content 

FC  Tip  Globe  ROI  IN 

NH  Tip  Globe  ROI  IN 

Pulling  needle 

NH  Unloaded 

NH  Loaded 

NH  Tip  Globe  ROI  OUT 

tying  knot 

FC  Globe  ROI  OUT 

FC  Grasping 

FCWalaadtfl 

FC  NH  Pulling  (needs  lo  be  defined) 

Cutting  thread 

SC  cutting  into  the  eye  ROI 

End  Suturing 

FC  Back  to  Tray 

k  !•..,* 

Scissor  Back  to  Tray 

SC  Tray  ROI  M 

NH  Back  to  tray 

NH  Tray  ROI  IN 

117 


During  Year  3,  we  initiated  a  process  to  begin  validation  of  the  gesture  recognition  algorithm,  through 
recruitment  of  expert  opthalmologists  and  interaction  with  Dr.  Paul  Latkany  of  the  Vision  Center  for 
Excellence.  The  software  was  installed  on  a  laptop  sent  to  Dr.  Latkany  to  allow  him  and  his  colleagues  to 
examine  the  data.  The  user  interface  was  altered  to  add  a  control  that  hides  the  event  tracks  in  order  to 
avoid  influencing  the  analysts  during  the  manual  gesture  entry.  Following  a  series  of  meetings  and 
conference  calls  where  the  different  teams  (SimGroup  and  Latkany' s  team)  agreed  on  the  modality  for 
the  validation  of  the  event  drive  gesture  segmentation  and  task  recognition,  a  series  changes  was  made 
to  the  software.  As  first  step,  a  table  of  all  possible  events  was  created,  showing  all  the  correlations 
between  the  different  low  level  events  combinations  and  the  corresponding  high  level  ones.  The 
package  sent  to  the  analysts  included  the  table  of  events  at  the  different  levels  (low  level,  high  level  1, 
subtasks  and  tasks).  Latkany's  team  would  perform  manual  gesture  recognition,  entering  a  keystroke 
each  time  a  specific  event  on  the  table  occurs  during  the  playback  of  previously  recorded  sessions  (user 
trials  at  the  USUHS  Ocular  Trauma  Courses).  The  manual  entries  on  a  subset  of  the  recorded  sessions 
were  to  be  compared  with  the  events  generated  by  the  automatic  system.  Manual  entry  would  be 
performed  only  for  the  high  level  events,  as  the  reliability  of  the  low  level  ones  is  very  close  to  the  ideal 
-  the  quantitative  identification  of  instrument  connection,  geometric  position,  etc.  do  not  have 
sufficient  statistical  noise  to  be  relevant.  High  level  events,  however,  derive  from  combinations  of  low 
level  events  (patterns  of  LLE)  which  may  vary  or  include  errors.  Validation  would  verify  the  reliability  of 
the  selected  patterns,  namely  the  logic  expressions  that  will  trigger  the  generation  of  a  HLE.  Once 
manual  entry  is  completed,  we  will  be  able  to  perform  a  comparison  between  the  two  sets  of  events  and 
calculate  the  reliability  of  the  event  driven  gesture  segmentation  and  task  recognition.  Due  to 
scheduling  difficulties,  completion  of  this  work  was  not  achieved  by  the  end  of  the  program. 

In  parallel  with  this  effort,  the  analysis  engine  reached  the  stage  of  development  that  allowed  the 
application  of  metrics  exclusively  to  motions  related  to  the  phases  of  a  procedure,  excluding  those  that 
are  not  relevant  (e.g.  simulator-related  events  such  as  connecting  instruments,  which  would  be 
extraneous  to  a  real  procedure).  For  example,  in  the  section  below,  manual  entry  indicates  the  start  and 
end  of  the  cleaning  phase,  while  the  automatic  system  further  selects  only  the  green  section  of  data  for 
consideration. 


1  2 
Begin  Cleaning 

End  Cleaning 


Figure  130:  partitioning  of  data  into  relevant  zones 

Further,  detectable  error  states  have  been  implemented  in  the  warnings  section  of  the  analysis  engine 
(Table  3). 
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Table  3:  Gesture  detection  system  warning  engine  events  -  extendable  list. 


Tasks 

Warning  Analysis  Engine 

Cleaning 

Improper  instrument  disconnection 

Timeout  for  beginning  of  cleaning 

Timeout  for  end  of  cleaning 

Improper  instrument  set  selection 

Exploration 

Improper  instrument  disconnection 

Improper  instrument  set  selection 

Timeout  for  beginning  of  exploration 

Timeout  for  end  of  exploration 

Grasping  tissue  with  excess  force 

Grasping  while  outside  of  proper  workspace 

Suturing 

Improper  instrument  disconnection 

Improper  instrument  set  selection 

Timeout  for  beginning  of  suturing 

Timeout  for  end  of  suturing 

Grasping  tissue  with  excess  force 

Grasping  while  outside  of  proper  workspace 

Cutting  suture  while  outside  proper  workspace 

Excess  distance  of  needle  motion  (dangerous  needle  pulling) 

2.3.7.c.v.  Simulation  Content  Frame  Structure 

Extending  the  content  frames  described  earlier,  during  Q7,  the  scripting  language  (XML-based) 
developed  to  manage  the  simulation  content  was  extended,  introducing  new  functionalities  and 
extending  the  syntax.  Included  in  the  updated  structure  describing  of  the  content  and  functionality  for  a 
given  node  in  the  procedure  were  user  interface  slider  button  positioning  and  extensions  to  the  types  of 
buttons  present  in  the  simulation  flow. 

To  support  good  coding  practice  and  simplify  development  for  future  users,  the  capability  to  add 
comments  in  the  frame  structure  was  added.  Further  replacement  of  the  system  condition  index 
(numbers  between  the  <system_Cond>  tags  -  see  code  extract  below)  with  a  set  of  mnemonic  labels 
was  implemented. 

//  first  slide  with  menu  selection 

<Section> 

<ID>1</ID> 

<Slide>Contents\Simulations\MMVR\Slides\2013MMVROCFGUI . 001 . jpg</Slide> 

<Audio>none</Audio> 

<Light>0</Light>  //  no  light  on  the  mannequin 

<Events> 

<Buttonl>0, 1, 0 . 0000, 0 . 4870</Buttonl>  //  NO  Button  on  the  left  side  of  the  screen 
<Button2>3, 23, 0 . 8339, 0 . 4870</Button2>  //  Next  Button  on  the  right  side  of  the  screen 

<Button3>0, 0,0.4169,0. 9000</Button3> 

<Time>0, 2</Time> 

<system_Cond>-l , 0</ system_Cond> 

<system_Cond>0 , 0</system_Cond> 
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<system_Cond>0 , 0</system_Cond> 

<system_Cond>0 , 0</system_Cond> 

<system_Cond>0 , 0</system_Cond> 

<Warning_Cond>0 , 0</Warning_Cond> 

<Warning_Cond>0 , 0</Warning_Cond> 

<Warning_Cond>0 , 0</Warning_Cond> 

</Events> 

</ Section> 

The  architecture  was  further  modified  by  separating  the  slide  content  management  and  the  gesture 
segmentation.  These  two  last  components  now  run  independently.  Some  of  the  Gesture  recognition 
routines  were  improved  to  increase  accuracy  and  tolerance  in  detection  of  incorrect  movements  in  the 
workspace. 


2.3.7.c.vi.  Content  Development  for  Scenarios 

Towards  the  end  of  Year  2,  the  didactic  content  for  the  eyelid  laceration  module  was  fully-developed  to 
incorporate  the  complete  set  of  steps  involved  from  a  clinical  point  of  view.  These  steps  were 
established  after  reviewing  existing  training  materials  as  well  as  from  several  meetings  with  our  subject 
matter  experts.  While  comprehensive  and  available  for  use  on  the  simulator,  for  the  Ocular  Trauma 
courses,  we  provided  a  subset  consistent  with  Dr.  Mazzoli's  teaching  requirements  at  the  2013  course, 
allowing  a  user  to  be  guided  by  an  instructor  and  immediately  begin  suturing. 
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Figure  131:  full  sequence  of  available  content  and  sequential  guidance  slides  for  lid  laceration  scenario. 

Of  the  61  screens  of  content,  such  as  that  pictured  below,  we  set  to  "hidden"  those  steps  that  would 
typically  be  involved  in  patient  prep  or  early  diagnosis. 
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Determine  if  okay  to 
add  eyedrops 


[PLAY  AUDIO  FILE]  Click  next  button  on  right. 


Figure  132:  Pre-scenario  preparation  informational  slide,  available  for  full  content  use  (hidden  for  2013  Ocular 
Trauma  Course  use) 

To  better  integrate  with  our  underlying  learning  architecture,  the  eyelid  laceration  didactic  content  was 
organized  as  an  individual  module.  As  the  system  is  designed  to  support  multiple  scenarios,  providing  an 
architecture  that  supports  future  content  is  important.  The  user  begins  by  selecting  the  module  he/she 
wants  to  train  on  and  clicks  the  start  button  in  the  center  of  the  screen.  Additional  modules  would  be 
found  by  navigating  to  the  left  or  right  of  the  central  area  on  the  screen. 


SELECT  TRAINING  MODULE 


[play  audio  pile]  Ability  to  scroll  through  modules  horizontally  using  arrows  on  left 
and  right.  Click  start  to  continue. 


Figure  133:  Scenario  selection  interface  screen 

Once  a  module  is  selected,  the  user  is  then  presented  with  the  ability  to  practice  a  specific  technique  to 
suturing  the  injured  eyelid.  We  found  that  each  surgeon  and  subject  matter  expert  we  consulted  had 
strong  opinions  about  the  "right"  way  to  provide  guidance  to  the  novice  trainee.  As  they  all  referenced  a 
"textbook"  way  of  suturing  as  an  underlying  foundation  to  their  personal  approach,  we  began  by 
implementing  that  basic  version. 
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SELECT  A  TECHNIQUE 
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THE  MAZZOU  TECHNIQUE  FREfTAt 

STANDARD  TECHNIQUE 

VWamua  ngittts  linu  vrl  augur  laurert 
rutrum  faucibtu  dolor  auctor.  Nullam  quis 
n*u»  tgrt  uma  mollis  urnan  vri  ru  Iro. 

Vfstibutum  id  lipili  porta  felts  ruismod 
srtnprr  Cn*  justo  odk).  dapibus  ar  fanlbis 
m.  rgestts  rgrt  qium 

Cras  mauls  conswtrtur  punis  sit  arort 
frrmrnlum.  Doner  srd  ndio  dui  Macrroas 
sed  diam  rgrt  risus  varius  blandit  sh  amrl 


tpfl  CEHEfLM  EkStDAI 


[PLAY  AUDIO  FILE]  Ability  to  scroll  through  modules  horizontally  using  arrows  on  left 
and  right.  Click  start  to  continue. 


Start 


Figure  134:  Selection  among  different  techniques  for  given  procedure 

Our  design  allows  a  novice  to  learn  the  nominal  approach  to  suturing  a  laceration  of  the  upper  eyelid. 
Once  completed,  the  user  is  then  able  to  practice  the  individual  techniques  advocated  by  different 
surgeons.  As  they  progress  through  the  scenario,  video  and  audio  commentary  can  accompany  a 
custom-tailored  set  of  didactic  content  in  the  interface  to  illustrate  to  the  novice  where  they  are 
deviating  from  the  textbook  procedure. 

2.3.8.  “Month  11”  -  Completion  of  prototype  stand-alone  ocular  and  craniofacial  trauma 
training  module,  control  software  supporting  sensing  and  response  functionality. 

During  Year  3,  the  hardware  of  the  standalone  system  reached  a  near  final  stage,  with  the  addition  of  a 
mounting  frame  that  allows  for  the  attachment  and  reconfiguration  of  a  touch-screen  interface  (to 
replace  the  prior  laptop-based  system),  and  a  four  degree  of  freedom  microscope  mounting  system  for 
positioning  the  AR  microscope  relative  to  the  eye  module. 

The  microscope  mount  is  designed  as  a  closed  loop,  5-bar  linkage,  with  a  pair  of  thumb  screws  to  either 
side  of  the  microscope  head  that  each  lock  two  degrees  of  freedom  of  the  linkage.  The  linkage  allows 
for  repositioning  of  the  microscope  head  in  the  horizontal  plane  above  the  mannequin  head  and 
moderate  rotation  of  the  microscope  (in  addition  to  that  available  by  rotating  the  microscope  within  the 
ring-mount).  The  ring-mount  provides  adjustment  of  the  vertical  (focus)  position  of  the  microscope.  As 
with  the  touch-screen  mount,  the  microscope  mount  is  modular,  allowing  for  quick  installation  and 
removal  without  modification  of  the  rest  of  the  system,  to  facilitate  rapid  transition  between 
oculoplastics  and  eye  globe  surgery  scenarios  (Figure  135). 
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Figure  135:  Standalone  system  with  modular  touch-screen  and  AR  microscope  components  mounted.  The  head 
form  remains  to  be  replaced  by  the  new  skeleton  and  full-face  versions  of  the  head  and  trauma  modules.  In  the 
left  image,  note  the  revised  instrument  tray,  which  now  does  not  include  the  suture  boxes  (as  recommended  by 
experts  at  USUHS)  and  the  somewhat  extended  utility  tray,  which  now  extends  closer  to  the  mannequin's  chin 
than  the  original  version  did. 

The  touch-screen  mount  is  modular  and  can  be  installed  either  to  the  right  or  left  of  the  stand  to 
accommodate  users  performing  procedures  on  the  left  or  right  of  the  mannequin,  using  only  two  large 
thumb  screws  (Figure  136). 


Figure  136:  Images  demonstrating  modularity  of  the  display  components;  Left  or  right  positioning  of  the  touch¬ 
screen,  attachment  or  removal  of  the  AR  microscope. 

Following  revisions  and  improvements  after  each  of  three  demonstrations/uses  at  the  annual  USUHS 
Ocular  Trauma  course,  the  system  design  and  software  have  reached  a  stable,  useable  form.  The 
standalone  platform  includes  the  modular  components  to  support  the  AR  microscope.  During  testing  of 
the  system,  it  was  found  that  the  weight  of  the  microscope  tended  to  make  the  system  prone  to  tipping 
towards  the  user  with  small  amounts  of  force,  so  a  revision  to  the  microscope  mount,  with  an  extension 
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towards  the  rear  of  the  system  and  steel  bar  counterweights,  was  designed  and  installed  in  the  final 
months  of  the  program  (Figure  137). 

The  new  version  of  the  head  and  neck  now  replace  the  original  head  form,  attached  to  the  frame  using  a 
new  adaptor  platform.  The  location  of  the  eyes  of  the  new  head  have  been  closely  aligned  with  those  of 
the  old  mannequin,  relative  to  the  magnetic  tracking  system  emitter,  minimizing  any  changes  to  the 
interface  software  that  depend  on  the  location  of  the  Regions  of  Interest  around  the  eyes.  The  newly 
acquired  additional  instruments  described  at  the  June  2014  IPR  meeting  were  installed  into  the  standard 
instrument  holder  structure,  and  a  final  revision  to  the  mini-circuit  board  for  the  holders  which 
simplifies  adjustment  of  gains,  offsets  and  instrument  ID  detection  was  designed  and  awaits  assembly 
into  the  full  suite  of  instruments. 


Figure  137:Original  head  form  and  standalone  structure;  New  head  form  with  adapter  mounting  structure  to 
attach  directly  to  the  standalone  frame.  Assembled  frame  with  new  head.  System  with  counterweights  for  AR 
microscope. 

In  addition  to  the  structure,  alternate  facial  modules,  supporting  training  for  retrobulbar  hemorrhage 
and  facial  fracture  were  prepared.  The  first  facial  model  shown  in  Figure  138  is  the  default,  healthy 
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tissue  version,  which  can  be  modified  by  manually  lacerating  the  lids.  The  face  including  forehead, 
tissues  forward  of  the  ears  and  the  neck  below  the  larynx  forms  one  full  module  that  is  attached  to  the 
cowl,  which  covers  the  sides  and  back  of  the  head,  and  the  neck  past  the  base  of  the  neck,  using  a  fine 
tooth  zipper  system.  The  cricothyroidotomy  site  has  been  left  as  a  separate  component  for  ease  of 
replacement.  In  the  final  iteration,  we  chose  to  create  unified  eye/face  trauma  modules,  rather  than  a 
series  of  distinct  components  to  be  assembled  on  site  with  gaps  and  margins  at  multiple  sites  about  the 
face.  The  gap  between  cowl  and  eye  module  of  the  old  version  of  the  head  form  could  never  be 
minimized  to  the  point  of  being  unremarkable,  and  was  present  directly  in  the  field  of  view  of  the 
trainee. 


Figure  138:  Base  healthy  face  attached  to  cowl.  Face  module  with  swollen  periocular  tissue.  Face  module  with 
massive  trauma  and  exposed  oral  cavity. 

2.3.9.  “Month  12”  -  Testing  of  fit,  function  of  sensors,  actuators,  replaceable  components. 

The  following  sections  review  testing  of  a  variety  of  the  animatronic  components  of  the  system 

2. 3. 9. a.  Air  muscle  and  neck  motion  testing  and  additional  neck  design 
Following  early  experiments  to  characterize  our  air  muscle  designs,  described  above,  during  Q5,  we 
completed  the  test  rig  to  allow  testing  of  motion  in  all  directions  and  reevaluate  the  capabilities  of  the 
new  muscles  (Figure  139).  This  system  appears  to  be  sufficient  to  generate  neck  motions  that  would  be 
used  in  the  mannequin-integrated  version  of  the  head  (Phase  III)  to  simulate  a  struggling/gagging 
casualty,  a  feature  that  does  not  appear  to  be  present  in  the  commercial  market. 
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Figure  139:  Motion  control  of  neck  using  combinations  of  single  or  adjacent  air  muscles.  Activation  of  single  air 
muscle  to  illustrate  range  of  motion.  Base  of  "neck"  at  the  bottom,  "head"  would  be  at  the  top. 

In  Q8  we  continued  this  work  in  designing  a  more  anatomically  correct  neck  structure  that  integrated 
the  neck  pitching  motions  with  head  yaw.  The  mannequin-integrated  version  of  the  system 
incorporates  the  active  motions,  while  the  stand-alone  version  retains  the  linkages  to  allow  the  surgeon 
to  passively  move  the  head  with  the  same,  normal  range  of  neck  motion,  but  without  the  actuation 
components,  thereby  minimizing  metallic  elements  that  can  distort  the  surgical  instrument  tracking 
system. 


Using  the  anatomical  models  that  were  segmented  during  Year  1  and  concepts  developed  during  our 
earlier  COMETS  autonomous  casualty  simulation  mannequin  project,  we  created  a  flexible  neck 
structure  using  a  series  of  vertebral  bodies  and  disks,  each  of  which  accommodates  a  fraction  of  the  full 
flexing  range  of  motion  of  the  human  neck. 


Figure  140:  Anatomically  inspired  neck  structure  CAD  model,  physical  implementation,  and  mounting  to  trauma 
mannequin. 
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The  vertebral  bodies  have  flanges  extending  away  from  their  cores  that  mate  with  the  McKibben 
muscles,  so  that  they  rotate  with  respect  to  each  other  when  the  head  turns.  A  central  flexible 
transmission  shaft  provides  tensile  strength  and  links  the  base  of  the  neck  to  a  head  rotation  motor  that 
will  be  mounted  in  the  head.  Human  neck  range  of  motion  is  different  in  flexion/extension  and  lateral 
abduction,  so  the  vertebral  bodies  were  designed  such  that  their  surface  contours  allow  for  more 
motion  before  one  body  collides  with  the  next  at  the  front  and  back  of  the  body  than  the  left/right 
direction. 


Figure  141:  Vertebral  body  and  disk  designs 

In  the  course  of  designing  the  neck  mechanism,  we  reviewed  the  McKibben  muscle  design.  The  earlier 
work  and  inquiry  showed  that  sparse  mesh  permits  larger  overall  deformation.  Thinner  bladder  walls, 
for  a  given  material  stiffness  allow  for  greater  deformation  for  a  given  supply  pressure,  as  does  a  softer 
material  for  a  given  thickness.  We  identified  both  thinner  wall  silicone  tubing  and  lower  durometer 
tubing  and  redesigned  the  muscles  to  accommodate  the  new  material.  In  addition,  we  found  that  the 
original  tube  fittings  used  as  attachment  points,  which  were  smaller  in  diameter  than  the  silicone  and 
mesh  tubing  diameters  resulted  in  pinching  at  the  muscle  ends,  reducing  the  available  amount  of  muscle 
contraction.  Related  to  that,  standard  hose  clamps  that  had  been  used  were  prone  to  pinch  and 
puncture  the  silicone  tubing  walls.  We  replaced  the  fittings  and  hose  clamps  with  larger  diameter 
fittings  and  plastic  snap-grip  hose  clamps  without  the  hard,  sharp  edges. 

With  these  design  changes,  we  reexamined  McKibben  muscle  performance,  comparing  the  thin  and  two 
grades  of  soft  rubber  to  determine  the  best  available  of  materials  sourced  from  industrial  supply  houses. 
The  thin  wall  silicone  was  found  to  have  the  best  performance  of  the  materials  tested. 
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Figure  142:  Retesting  McKibben  muscles  with  new  materials,  fittings.  500g  load  results  shown:  relative  contraction 
for  black  latex  (blue),  amber  latex  (red)  and  thin  wall  silicone  tubing  (green) 

A  mating  platform  (shown  extending  below  cowl  in  Figure  138)  was  designed  to  attach  the  neck  design 
to  the  Laerdal  SimMan  Essential  mannequin  that  we  acquired  in  Q6.  Using  the  pneumatic  control 
system  described  above,  we  tested  the  range  of  neck  motion.  While  the  neck  assembly  was  capable  of 
the  full  human  range  of  motion,  when  coupled  to  the  air  muscles,  the  available  contraction  appeared  to 
reduce  the  actual  range  delivered.  In  addition  while  the  neck  flexes  quite  far  when  the  muscles 
contract,  the  platform  shown  (representing  the  mounting  structure  of  the  skull)  may  pitch  by  a  lesser 
amount.  The  mannequin  therefore  moves  less  under  its  own  control  than  can  be  generated  by  manual 
manipulation  of  the  head/neck  by  a  trainee.  In  future  work,  we  will  endeavor  to  increase  the 
controllable  motion,  however  for  the  purposes  of  creating  a  system  that  can  present  autonomous 
motion  that  will  indicate  consciousness  and  present  a  simulation  of  a  casualty  resisting  treatment  (e.g. 
intubation/cricrothyroidotomy),  this  system  satisfies  the  basic  goals. 
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Figure  143:lnitial  range  of  motion  of  neck  actuation  system 


Figure  144:  Manual  manipulation  of  neck  with  attached  skull  model 

The  current  range  of  rotational  motion  of  the  head  with  respect  to  the  neck  is  +/-  55  degrees,  similar  to 
that  of  the  human  head  relative  to  the  upper  segments  of  the  cervical  spine.  Early  testing  performed 
without  the  cowl  and  face  installed  suggests  that  the  existing  motor  may  not  be  able  to  provide 
sufficient  torque  to  drive  the  system  through  this  full  range,  however  an  alternative  design  is  available 
should  the  full  range  be  desirable  later  (e.g.  for  commercial  partner  development).  The  jaw  motion  is 
similarly  nearly  the  full  range  of  angles  (30  degrees),  with  the  final  value  to  be  determined  once  the 
facial  component  is  installed. 
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In  Year  3,  we  examined  the  detailed  placement  of  the  anchor  points  for  the  air  muscles;  while  there  is 
limited  real  estate  for  the  upper  ends  of  the  muscles,  the  neck  opening  of  the  mannequin  allows  for 
both  wide  and  narrow  positioning.  In  principle,  a  wider  stance  should  provide  for  a  higher  activation 
torque  when  flexing  the  neck,  while  a  narrower  stance,  with  the  muscle  closer  to  the  axis  of  the  neck, 
should  result  in  a  larger  range  of  motion.  Findings  were  mixed,  with  the  expected  results  occurring  in 
the  back  and  left  directions,  the  reverse  to  the  right,  and  no  noticeable  effect  towards  the  front.  As 
such,  the  narrow  footprint  configuration  will  be  used,  to  conserve  space  for  other  elements  within  the 
neck. 

During  Year  3,  we  extended  this  work  to  add  anatomical  features  of  the  spinous  processes,  with  fracture 
elements  included.  These  are  designed  to  be  palpable  landmarks  and  indicative  of  potential  spinal 
injury.  A  palpation  test  object  was  created  and  reviewed  by  our  local  SMEs  to  provide  feedback  on 
whether  the  choices  of  skin  and  muscle  substitute  materials  allowed  for  a  realistic  tactile  sensation. 


Figure  145:  Neck  element  design  with  addition  of  anatomical  features  to  the  common  core.  Palpation  test  module, 
including  neck  surface  contour  based  on  CT  anatomy  with  embedded  vertebral  body  and  fractured  spinous 
process. 

2.3.9. b.  Blink  actuation  testing 

In  Q7,  much  progress  was  made  on  the  blinking  mechanism.  Based  on  developments  described  above,  a 
prototype  blinking  portal  was  created  which  demonstrated  an  effective  solution  using  embedded  nylon 
string  in  the  tarsal  plate  which  can  then  be  actuated  by  a  servo.  The  string  can  be  physically  attached  to 
the  substrate  in  future  iterations.  This  solution  is  consistent  with  the  earlier  developments  of  the  servo- 
driven  blink  mechanism. 
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Testing  revealed  that  changes  to  the  portal  were  necessary,  consisting  of  incising  relief  flaps  into  the 
silicone.  These  flaps  allow  the  silicone  to  bend  more  freely  and  control  the  way  in  which  it  deforms.  In 
production,  these  flaps  will  be  incorporated  into  the  mold  design.  This  feature  addresses  one  of  Dr. 
Mazzoli's  principal  suggestions  regarding  the  eyelids,  namely  that  redundant  tissue  should  be  present, 
permitting  more  realistic  manipulation  and  proper  creasing  when  the  eyelid  blinks. 


Figure  146:  Servo  control  of  module  eyelid  blinking 

2.3.10.  “Month  14”  -  Design  of  modifications  to  trauma  mannequin  for  integration  of  ocular 
and  craniofacial  simulator  structure  module. 

This  program  conceived  that  the  eye/face  trauma  simulator  would  be  integrated  into  a  full-body  medical 
simulation  mannequin,  ideally  a  commercial  system,  or  into  our  prior  COMETS  autonomous  training 
system  mannequin  as  necessary.  During  year  1,  we  conducted  a  series  of  examinations  of  a  cross 
section  of  commercial  mannequins  including: 

•  Laerdal  SimMan  at  the  Harvard  Center  for  Medical  Simulation 

•  Gaumard  Hal  at  the  Harvard  CMS 

•  A  prototype  of  the  CAE-Healthcare  Caesar  mannequin,  while  on  display  at  the  MGH  Learning 
Laboratory  facility,  which  is  the  core  of  MGH's  simulation-based  training  efforts 

•  A  CAE/METI  iStan  mannequin  at  the  Tufts  University  School  of  Medicine-Clinical  Skills  and 
Simulation  Center 

•  A  CAE/METI  METIMan,  which  was  briefly  on  loan  to  us  specifically  for  examination 
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•  A  Laerdal  Nursing  Kelly  system  used  by  one  of  our  team  (Bardsley)  for  an  unrelated  project 

In  addition,  one  of  our  team  (Ottensmeyer)  had  a  parallel  project  to  modify  the  head  of  a  Laerdal  ALS 
simulator  to  add  functionality,  allowing  us  to  closely  examine  the  structure,  especially  the  airway 
components,  of  this  mannequin  head/neck  module. 

Based  on  the  analysis,  we  initially  considered  acquiring  a  Laerdal  Kelly  mannequin  for  test  integration. 
This  lower  cost,  lower  fidelity  mannequin  than  planned  for  the  final  system  would  allow  us  to  "walk 
before  we  run"  and  allow  for  additional  space  within  the  physical  structure  of  the  mannequin  for 
necessary  actuator,  reservoir,  power  and  other  systems.  Efforts  towards  development  of  the 
standalone  system,  however,  took  early  priority,  so  we  later  moved  directly  to  a  higher  fidelity  system. 

During  Q5  we  discussed  the  availability  of  CAE  Healthcare's  Caesar  mannequin,  which  was  not  expected 
to  be  available  for  purchase  during  our  original  timeline.  At  the  MHSRS  2012  conference,  one  of  the 
mannequins  was  on  display  and  the  sales  staff  conveyed  that  they  were  available  for  distribution. 
Examination  of  the  system,  however,  suggested  that  integration  of  a  new  head  onto  the  mannequin 
would  be  more  difficult  than  the  options  we  had  been  examining  with  Laerdal.  In  addition,  the  Laerdal 
system  that  had  sufficient  capabilities  for  our  purposes  costs  approximately  half  as  much  as  Caesar.  In 
addition,  an  equivalent  of  Laerdal's  SDK  was  not  available  for  Caesar.  As  a  result,  we  submitted  a 
purchase  requisition  for  a  Laerdal  SimMan  Essential,  which  arrived  during  Q6. 

In  Q6,  the  early  work  involved  opening  the  mannequin  to  identify  power  and  air  pressure  supplies,  test 
its  normal  functionality  under  control  of  the  Laerdal  software,  and  begin  a  detailed  look  into  the  SDK. 
Laerdal  provided  technical  details  of  the  mannequin's  structure,  pneumatic  supply  and  electrical  and 
communications  capabilities. 

We  combined  the  CAD  models  of  the  main  internal  structural  plate  from  Laerdal  with  a  3D  laser  scan  of 
the  outer  surfaces  of  the  mannequin  shoulders,  using  a  long  focal  distance  photo  of  the  mannequin  to 
aid  in  aligning  the  head/neck  anatomy  with  the  original  mannequin  head.  This  is  serving  as  the  basis  for 
our  structural  design  of  the  connection  and  mounting  of  our  head  to  the  mannequin's  body. 

As  described  earlier,  we  designed  for  use  of  the  mannequin's  air  and  power  supply  to  drive  the 
mechanisms  for  the  head  and  neck. 

Immediately  inside  the  mannequin's  left  shoulder  are  a  series  of  electrical  connectors,  one  of  which  is 
dedicated  for  head  control.  It  includes  terminals  for  12V  supply  and  a  serial  communications  port  that 
commands  the  standard  mannequin  head.  We  added  our  own  connector  in  place  of  the  original  to 
access  power  and  detect  the  pulse-control  signals  of  the  mannequin,  so  that  the  hemorrhage  system 
could  be  synchronized  with  the  mannequin's  pulse.  This  was  used  to  power  the  Arduino  Mega  and  the 
series  of  pneumatic  latching  valves  described  in  section  2.3.5.a.v.  As  described  earlier,  in  addition  to  the 
control  of  the  valves,  the  Arduino's  software  was  expanded  to  control  up  to  12  servos  (including  the 
eight  designed  into  the  blinking/eye  motion/proptosis  mechanism),  two  electronically  controlled 
pressure  regulators,  up  to  10  PWM-controlled  devices  (e.g.  fluid  or  air  pumps)  or  other  digitally 
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controlled  devices  (e.g.  additional  valves  for  hemorrhage),  up  to  two  stepper  motors  (head  rotation), 
measurement  of  16  analog  sensors. 

The  pneumatic  supply  provided  pressure  that  varied  from  a  peak  of  approximately  1  atmosphere  and 
fell  by  approximately  25%  before  the  mannequin's  compressor  turned  on  to  recharge  the  internal 
reservoir.  This  pressure  drop  is  related  to  the  mannequin's  own  breathing  mechanism,  which  consumes 
air  during  each  breath  (small  internal  bladders  push  the  main  chest  plate  outwards),  allowing  the  system 
pressure  to  fall  since  there  is  no  regulated  internal  reservoir.  Our  testing  of  the  neck  mechanism  under 
this  air  supply  showed  that  with  the  fall  of  system  pressure,  actuated  air  muscles  lost  force  cyclically  as 
the  mannequin  compressor  cycled,  with  slowly  changing  neck  position  as  pressure  fell,  and  a  return  to 
the  set  position  when  the  compressor  ran.  For  the  prototype,  this  ongoing  variation  in  neck  position 
was  considered  to  be  a  useful  indication  of  consciousness. 


Figure  147:  Early  test  of  air  muscle  system  integrated  with  mannequin  air  and  power  supply.  Blue  tubing  is 
connected  in  place  of  stock  head  air  connector,  twisted  pair  (black/red)  wires  supply  mannequin  power  to  the 
solenoid  valve  control  circuitry  and  the  Arduino  Mega  microcontroller. 

During  Q7,  we  isolated  one  of  the  problems  that  appeared  during  discrete  component  testing:  while  the 
air  muscle  valves  are  specified  for  use  with  12V  control  signals,  and  the  mannequin's  supply  voltage  is 
also  12V,  the  transistors  that  switch  the  valves  on  and  off  have  their  own  voltage  drops  in  series  with  the 
valves.  As  a  result,  the  voltage  available  to  the  valves  is  reduced  below  12V,  by  an  amount  that  makes 
valve  activation  unreliable.  Two  solutions  were  identified:  replace  the  12V  valves  with  their  6V 
equivalents,  or  use  a  voltage  converter  which  boosts  the  12V  supply  up  to  15V  (at  the  cost  of  drawing 
more  current  from  the  batteries),  so  after  the  transistor  voltage  drop,  the  valves  would  still  have  enough 
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driving  force  to  reliably  activate.  A  suitable  voltage  converter  was  ordered,  received  and  tested  to 
confirm  that  it  functioned  as  planned. 

During  Q8,  in  conjunction  with  the  neck  mechanism  developments  described  above,  we  used  the  CAD 
models  provided  by  Laerdal  for  the  main  frame  of  the  mannequin  to  create  a  mounting  frame  that 
mates  the  neck  to  the  mannequin  main  body.  The  attached  neck  is  shown  in  Figure  143  and  Figure  144. 

The  upper  end  of  the  neck  is  anchored  to  the  neck  and  bends  on  it  using  an  integrated  vertebral  disk 
structure.  The  lower  end  is  fastened  to  a  custom  mounting  plate  that  mates  with  the  Laerdal 
mannequin  frame.  This  plate  can  be  replaced  with  alternate  structures  to  make  the  system  physically 
compatible  with  other  mannequin  bodies. 


Figure  148:  Skull  attachment  design  features  for  the  neck,  accommodating  anchor  sites  for  the  ends  of  the 
McKibben  muscles;  neck  system  design  as  mounted  to  Laerdal  structural  frame,  with  CT  anatomy  spine  overlaid; 
CD  model  including  skull  to  verify  proper  scaling  and  size  of  neck  components. 

Based  on  our  findings,  we  determined  that  with  the  exception  of  removal  of  the  mannequin's  stock 
head  and  disconnection  of  power  and  fluid  lines,  no  further  physical  modifications  to  the  mannequin 
were  necessary  to  accommodate  our  system.  The  spaces  in  the  abdomen  and  the  chest  were  sufficient 
to  fit  the  additional  valves,  electronics  and  fluid  reservoirs  necessary  for  integration  of  the  head/neck 
into  the  Laerdal  body. 
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Figure  149:  Disassembly  and  examination  of  Laerdal  SimMan  Essential  mannequin  in  support  of  designing  interface 
components  to  mate  the  new  head/neck  to  the  commercial  body.  Aligned  CAD  models  in  use  for  design  of 
attachments  between  new  head/neck  and  stock  mannequin. 

2.3.11.  “Month  16”  -  Revision  of  physiology  model  software. 

The  selection  of  a  mannequin  platform  substantially  drove  the  approach  to  physiological  integration 
between  the  eye/face  trauma  system  and  the  mannequin.  The  Laerdal  SimMan  Essential  standard 
control  interface  at  the  time  of  acquisition  provided  detailed  controls  for  all  of  the  available 
physiological  parameters  and  mannequin  actuators.  As  such,  it  has  no  independent  physiological  model 
driving  its  behavior;  the  instructor/simulator  technician  actively  controls,  for  example,  changes  in  blood 
pressure  (which  may  be  set  for  step  or  ramp  changes),  but  blood  loss  or  trainee  intervention  do  not 
directly  affect  mannequin  physiology.  This  was  distinct  from  the  more  expensive  CAE/METI  systems,  for 
example,  which  had  an  underlying  physiological  model. 

In  Year  1,  in  advance  of  receipt  of  the  physical  mannequin,  we  obtained  access  to  the  Laerdal  application 
programmer's  interface  software  tools,  which  Laerdal  provides  to  developers  so  that  they  can  develop 
control  and  feedback  tools  for  their  advanced  mannequins  (e.g.  SimMan).  Discussions  with  Laerdal's  Jim 
Azukas  showed  that  their  lower  cost  systems,  some  of  which  were  compatible  with  their  new  SimPad 
technology,  would  not  have  an  API  available  in  the  near  term,  so  use  of  such  mannequins  may  have 
permitted  integration  with  power  and  fluidic  systems,  but  control  and  physiological  modeling  would 
have  to  have  been  implemented  separate  from,  rather  than  integrated  with  the  mannequin's  own 
systems. 
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At  that  time,  we  also  examined  open  source  physiology  models,  such  as  Hummod  [Hester  et  al.,  2011]  in 
the  context  of  preparing  a  proposal  for  the  JPC-1  "Developer  Tools  for  Medical  Education  Public 
Physiology  Research  Platform"  program  announcement,  and  discussed  physiological  interactions 
primarily  related  to  eye  trauma/surgery  with  our  SMEs  Mazzoli  and  Freitag.  In  relation  to  performing 
eye  surgery,  they  identified  only  one  characteristic  physiological  effect  (beyond  those  related  to  blood 
loss  or  airway  obstruction),  namely  the  oculocardiac  reflex.  This  reflex  is  triggered  upon  compression  of 
the  eyeball  or  traction  applied  to  intraocular  muscles,  and  results  in  a  decrease  in  pulse  rate.  Given  the 
direct  control  of  the  Laerdal  mannequin  that  was  selected,  this  could  be  implemented  by  either 
presenting  the  operator  with  the  magnitude  of  force  applied  to  the  globe  (the  scope  of  this  simulator 
did  not  extend  to  inclusion  of  intraocular  muscles)  and  relying  on  the  operator  to  adjust  heart  rate,  or  a 
semi-automatic  system,  in  which  elevation  of  globe  force  would  generate  a  command  to  decrease 
mannequin  heart  rate. 

This  work  began  substantially  during  Q7.  In  the  first  two  months  of  the  quarter,  we  examined  the 
Laerdal  SDK  in  detail,  starting  with  familiarization  with  the  programming  language  (C#)  and  with  the 
layout  of  the  VS  (Visual  Studio)  C#  project.  To  avoid  potentially  damaging  the  mannequin,  when 
necessary  we  performed  preliminary  testing  of  code  using  the  Laerdal  virtual  mannequin  system,  which 
presents  a  system  that  responds  to  code  written  using  the  SDK  the  same  way  the  real  SimMan  would. 

The  initial  focus  was  the  LaerdalSimulatorAPI.dll  dynamically  linked  library.  The  goal  of  the  work  was  to 
implement  an  appropriate  approach  to  use  the  functions  and  methods  provided  by  the  dynamic  library, 
in  order  to  use  them  in  a  new  C++  application.  As  the  source  code  and  documentation  of  the  DLL  was 
not  provided,  extrapolation  of  the  content  of  the  DLL  was  necessary.  To  help  with  that  we  used  the 
"Object  Browser"  VS  functionality  and  the  free  .NET  decompiler  and  assembly  browser  "JetBrains 
dotPeek  1.0" 6. 

Early  approaches  aimed  at  minimizing  the  number  of  additional  files  needed  while  retaining  as  much 
simplicity  as  possible  in  the  conversion  the  functionality  available  in  the  C#-based  SDK,  to  functions 
compatible  with  our  C++-based  simulator  were  unsuccessful.  A  subsequent  approach  treating  each 
component  of  the  C#  SDK  separately,  creating  a  large  library  of  files  to  form  a  bridge  between  the  two 
languages  was  attempted  by  one  of  our  interns  next. 

2.3.11. a.  Developing  the  Wrapper  for  LaerdalSimulator.dll 
The  DLL  wrapper  works  as  a  bridge,  linking  the  C#  DLL  with  the  C++  client  application.  In  order  to  create 
the  link,  every  member  (classes,  delegates,  interfaces  and  enums)  of  the  C#  DLL,  had  to  be  re-declared 
inside  the  wrapper. 

To  create  the  wrapper  we  used  an  approach  based  on  creating  a  different  C++  file  for  every  class 
implemented  in  the  original  DLL,  and  another  single  C++  file  which  contains  enumerators  and  interface 
classes.  Because  of  that,  the  wrapper  project  consists  in  the  following  files: 


6  Download  and  documentation  available  at  www.jetbrains.com/decompiler 
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-  ExceptionData_Wrapper.h,  which  contains  the  declaration  of  the  class  ExceptionData_Wrapper 
and  its  methods/constructors/destructors 

-  ExceptionData_Wrapper.cpp  which  contains  the  implementation  of  the  methods  of  the  class 
Exception  Data_Wrapper 

-  ManikinException_Wrapper.h  which  contains  the  declaration  of  the  class 
ManikinException_Wrapper  and  its  methods/constructors/destructors 

-  ManikinException_Wrapper.cpp  which  contains  the  implementation  of  the  methods  of  the 
class  ManikinException_Wrapper 

-  ManikinFactory_Wrapper.h  which  contains  the  declaration  of  the  class 
ManikinFactory_Wrapper  and  its  methods/constructors/destructors 

-  ManikinFactory_Wrapper.cpp  which  contains  the  implementation  of  the  methods  of  the  class 
ManikinFactory_Wrapper 

-  LaerdalSimulatorAPI_Wrapper.h  which  contains  the  declaration  of  interface  classes,  delegates 
and  enumerators. 

-  LaerdalSimulatorAPI_Wrapper.cpp  which  contains  the  declaration  of  the  enums  (as  they  ore 
declare  already  in  the  .hfile  maybe  I  would  delete  them  from  here)  and  would  contain  (still  to  be 
done)  the  implementation  for  some  get/set  methods  for  the  interfaces. 

-  LaerdalSimulatorAPl.h  header  file  of  the  original  DLL  created  by  me  (need  some  changes). 

To  test  the  wrapper  our  CS  intern  created  a  small  Dialog  application  that  showed  the  client  a  window 
and  performed  a  series  of  functions  based  on  activation  of  dialog  buttons. 

The  first  test  performed  was  on  the  class  ExceptionData_Wrapper.  It  contained  a  series  of  properties 
(data)  that  could  be  read  and  written.  This  simple  test  did  not  need  require  the  protection  of  linking  to 
the  virtual  mannequin  and  was  performed  directly  on  the  physical  mannequin.  The  testing  procedure 
comprised  the  following  steps: 

•  calling  the  set  method  of  the  property  declared  inside  the  wrapper.  This  is  linked  to  the  set 
method  of  the  C#  DLL,  so  whenever  a  new  value  is  passed  to  the  wrapper  set  function,  the  same 
value  would  be  passed  to  the  property  of  the  LaerdalSimulator.dll. 

•  calling  the  get  method  declared  inside  the  wrapper.  Because  of  the  linkage,  this  would  read  the 
value  of  the  property  contained  in  the  C#  DLL 

•  if  the  two  value  are  the  same,  the  linkage  is  verified 

The  test  demonstrated  that  a  link  was  built,  a  first  step  towards  establishing  communication  between 
our  application  and  the  mannequin.  Unknown  to  us  at  that  time  was  that  there  was  a  mismatch  in 
firmware  versions  between  the  SDK  and  the  purchased  mannequin  that  slowed  further  progress  until 
we  found  and  resolved  this  issue. 

In  Year  3,  we  evaluated  LLEAP,  a  new  version  of  the  SimMan  SDK  that  became  available.  As  it  was  still 
under  pre-release  development,  we  determined  that  we  would  continue  to  use  the  older  SDK  and 
consider  a  shift  to  the  newer  technology  in  future  work. 
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Over  the  remainder  of  the  year,  we  developed  our  final  solution  for  linking  our  software  with  the 
mannequin  to  allow  the  control  and  querying  of  various  physiological  parameters.  We  established 
access  the  heart  rate  and  the  pulse-oximeter  functionalities,  which  are  representative  of  all  of  the  other 
accessible  parameters  of  the  mannequin.  As  the  SimMan  Essential  has  no  internal  physiology  model, 
control  of  its  parameters  by  our  system  is  equivalent  to  control  of  the  same  parameters  from  the 
instructor's  laptop,  essentially  completing  the  conceptual  component  of  integration  of  our  system  with 
the  controls  of  the  mannequin. 


Figure  150:  Sample  interface  showing  connection  between  software  and  mannequin,  a  text  box  where  the 
mannequin's  heart  rate  can  be  sampled  (from  the  instructor  laptop)  and  set  (overriding  instructor  laptop  settings), 
and  the  pulse-oximeter  sensor  can  be  turned  on  and  off.  The  full  suite  of  SDK-linked  physiology  parameters  is 
accessible  using  the  same  technique. 

We  made  additional  progress  in  interfacing  with  the  physiology  of  the  mannequin.  We  can  control  all 
parameters  that  can  be  externally  overridden,  and  read  the  mannequin's  physiology  for  the  remaining 
parameters  that  are  set  internally.  The  window  below  shows  the  WiFi  connection  parameters,  samples 
of  the  mannequin's  parameters,  and  control  over  both  binary  (on/off)  controls  and  analog  values  (e.g. 
heart  rate).  The  actual  application  of  these  parameters  will  be  internal  to  the  simulator  and  its  interface. 
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Figure  151:  (Left)  Parameter  control  interface  panel  (Right)  Display  from  Laerdal  mannequin  showing  Heart  Rate 
matching  set  value  from  control  interface  panel  (80bpm) 

2.3.12.  “Month  18”- 

2.3. 12. a.  Completion  of  prototype  integrated  ocular  and  craniofacial  trauma  system  with 
mannequin. 

During  Year  3,  we  completed  mounting  of  the  head  form  to  the  mannequin's  body,  primarily  for  fit 
testing  and  to  confirm  clearances  between  Laerdal  and  OCF  components.  All  of  the  functionality  of  the 
system  had  been  previously  established.  Examples  of  such  instances  show  the  OCF  head  and  the  stock 
component  laid  next  to  each  other,  with  the  OCF  head  attached  to  the  mannequin's  body,  and  the  final 
version  with  the  swollen  periocular  tissue  face  module  attached.  The  "cowl"  fits  within  the  standard 
opening  of  the  mannequin  neck. 


Figure  152:  Head/neck  with  Year  3  partial  facial  module  overlaid,  attached  to  Laerdal  SimMan  Essential.  Original 
SimMan  head  shown  for  references.  Final  version  of  head/neck  form  mounted  to  mannequin  with  swollen 
periocular  tissue  face  module  attached. 
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2.3.12.b.  Completion  of  augmented  reality  system  supporting  stand-alone  simulator. 

The  system  that  was  created  through  this  project  provides  for  a  wide  variety  of  modes  for  interacting 
with  the  artificial  anatomy,  beginning  with  conventional  video  display  and  touch  screen  control,  and 
extending  through  position-tracked  instruments  that  trigger  events  when  they  cross  into  geometric  or 
actuation/force  spaces,  instrument-free  hand-tracking  and  interaction  with  virtual  buttons  and  virtual 
moulage  projected  onto  the  scene,  to  the  video-augmented  binocular  surgical  microscope.  Computer 
controlled  lighting  is  automatically  brightened  to  direct  attention  towards  the  face,  or  dimmed  to  cue 
recognition  that  relevant  content  is  being  displayed  on-screen.  Content  development  was  focused 
primarily  on  the  needs  of  the  trainees  at  the  USUHS  Ocular  Trauma  course  learning  basic  eyelid 
laceration  repair,  but  "hooks"  into  all  of  the  functionality  of  the  system  are  available  for  implementation 
of  additional  scenarios,  from  directly  viewed  oculoplastics  to  microscope  mediated  corneo-scleral 
procedures,  and  anatomical  components  were  developed  to  address  nearly  all  of  the  originally  planned 
procedures. 


Figure  153:  Standalone  system  with  full  suite  of  AR  components  installed:  Kinect  operating  through  mirror  to 
enable  hand  tracking;  projector  displaying  virtual  buttons  for  non-contact  interaction  (illumination  projected  on 
table-top  and  face);  AR  microscope  with  position  tracking  to  enable  synchronized  motion  of  microscope  and 
imagery  overlaid  on  direct  view  of  facial  anatomy. 

2.3.13.  “Month  20”  -  User  testing  of  combined  system. 

This  phase  of  testing  was  performed  primarily  in  the  context  of  receiving  ongoing  feedback  on  the 
system  elements  from  the  SMEs  on  partially  assembled  components;  we  were  unable  to  schedule  a 
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dedicated  field  test  of  the  head-form  mounted  to  the  mannequin  together  with  the  demonstration/use 
needs  of  the  system  for  Ocular  Trauma  Course. 

The  SMEs  (Mazzoli  and  Freitag,  primarily,  as  well  as  the  faculty  at  the  USUHS  Course  who  used  the 
standalone  simulator)  found  the  increasing  realism  of  the  system  to  be  improvements  that  would 
support  better  training  of  their  trainees. 

The  mobility  of  the  head  and  neck  structure  was  acceptable  for  the  needs  of  the  ophthalmologist. 
Additional  feedback  from  the  first  responder  community  will  be  sought  in  future  efforts. 

2.3.14.  “Month  23”  -  Completion  of  revisions  based  on  user  testing. 

Final  identified  revisions  and  elements  for  completion  that  implemented  as  the  end  of  the  final 
extension  period  drew  to  a  close  included: 

•  Addition  of  the  final  port-expander  chips  to  the  controller  of  the  fluid  and  pneumatic  valve 
manifolds 

•  Creation  of  the  instrument  holders  for  the  additional  instruments  requested  for  the  2015  USUHS 
course  (described  in  section  2.3.4.c) 

•  Completion  of  the  counterweight  system  to  support  the  AR  microscope,  for  demonstration  at 
IMSH  2015 

•  Completion  of  the  mounting  frame  to  connect  the  mannequin-compatible  version  of  the 
head/neck  to  the  base  of  the  standalone  simulator 

•  Adjustment  of  microscope  optics  to  verify  proper  rotation  of  the  graphical  images  in  each 
eyepiece 

•  Fabrication  of  final  face  modules  for  demonstration  at  IMSH  2015 

2.3.15.  “Month  24”  -  Final  report  submission. 

This  report  represents  the  product  of  this  final  milestone. 
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3.  Key  Research  Accomplishments 

•  Creation  of  advanced  simulator  for  eye  trauma  based  on  proof-of-principle  demo  system 

o  Design  and  fabrication  of  interchangeable,  hot-swappable,  auto-identifying,  precisely 
registered  suite  of  sensorized  ophthalmological  instruments  that  can  be  extended  to 
additional  surgical  procedures 

o  Development  of  basic  data  acquisition  and  global  scoring  system  into  dynamically 
adaptable  platform  with  capacity  for  multiple  position  tracking  technologies  (magnetic, 
optical,  other)  and  definition  of  surgically  relevant  regions  of  interest 

o  Construction  of  modular  structural  framework  to  provide  mounting  of  and  registration 
between  mannequin  head,  instrument  tracking  system,  projection  display,  Kinect 
motion  tracker  and  AR  microscope,  novel  instrument/rack  interface  for  one-handed 
instrument  connector  interchange 

o  Development  of  binocular  operating  microscope  with  augmented  reality  capability: 
graphical  overlays  registered  and  dynamically  updated  to  track  real  anatomy 

o  Creation  of  software  tools  to  support  multiple  modes  of  AR  within  common  simulation 
platform:  projected  interfaces  controls  with  hand/instrument  tracking,  generation  of 
stereoscopic  imagery  and  alignment  tools  for  head-down  AR  interaction  through 
microscope 

o  Multiple  generations  of  anatomical  improvements  from  single  layer  left  eyelid  and 
simplified  globe  to  full-facial  trauma  modules  supporting:  simple  lid  laceration; 
canalicular  involvement/intervention;  retrobulbar  hemorrhage  indications/intervention, 
palpation  of  facial  fracture;  realistic  airway  features;  globe  with  anterior  and  partial 
posterior  chambers,  transparent  lens,  anatomically  thin  and  operable  cornea  and  sclera 

o  Preparation  of  software  implementation  for  validation  of  gesture/task  detection  by 
expert  observers,  provision  of  the  software  to  collaborators 

•  Development  of  surgical  gesture  sequence  recognition  algorithm  for  online,  real  time  tracking  of 
performance 

o  Conceptualizing  the  hierarchical  structure  of  surgical  procedures  from  high  level  tasks  to 
elementary  events  that  can  reliably  be  detected  through  tracking  of  instrument 
motions,  information  on  instrument  connectivity  and  force/position  data  from  strain 
gages  or  other  sensor  inputs 

o  Developing  state  space  structure  to  map  possible  transitions  between  states  generated 
by  individual  surgical  gestures,  sequences  of  gestures  and  sequences  of  sequences 

o  Recognition  of  reduced  computational  load  of  analysis  of  surgical  motions,  sub-tasks, 
tasks,  procedures  through  the  implementation  of  the  event-driven  surgical  gesture 
recognition 

o  Creation  of  streaming/video  data  acquisition,  storage  and  review  software  tools  to 
enable  expert  review  of  observed  behaviors  in  support  of  identifying  scenario-specific 
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gesture  sequences,  verification  of  automated  gesture  recognition  and  validation  of 
trainee/expert  performance  differences 

o  Improvements/expansion  of  software  code  and  gesture  recognition  architecture  to 
allow  for  post-hoc  analysis  of  recorded  data;  addition  of  manually  entered  and 
automatically  generated  event  flags  to  be  displayed  on  timeline  playback  system; 
implementation  of  latest  task  decomposition  and  warning  engine  creation 

•  Development  of  fabrication  techniques  for  realistic  anatomical  model  creation  and  development 
of  novel  sensing/actuation  systems  for  an  animatronic  surgical  head/neck  structure 

o  Segmentation  of  full  head,  neck,  upper  torso  high  resolution  CT  data  set  to  provide 
reference  3D  models  of  bone,  skin  surface,  muscle  zones  for  this  effort  and 
ongoing/future  developments  of  this  and  related  simulators 
o  Evaluation  of  alternate  fabrication  methods  for  lid  laceration  module  for  improved 
efficiency,  quality 

o  Development  of  sequential/interchangeable  mold  structures  for  creating  bespoke  facial 
modules  that  support  full-featured  through  simplified  module  fabrication,  reusing 
common  mold  components 

o  Addition  of  tarsal  plate  structure,  prototype  auxiliary  features  including  eye  lashes, 
brows,  grey  line,  Meibomian  gland  external  pores  as  available  options 
o  Improvement  of  eye  globe  modules,  to  include  improved  resolution,  transparent 
cornea,  pigmented  iris,  improved  geometry  sclera.  Evaluation  of  alternate  vitreous 
humor  simulant  materials 

o  Ongoing  improvements  to  anatomical  landmarks/features,  fabrication,  materials 
selection  for  periorbital  skin,  eye  lids,  canaliculi,  conjunctiva, 
o  Design  and  fabrication  of  jaw  motion  and  head  left/right  motion  systems 
o  Design  and  fabrication  of  neck  structure  with  anatomical  features  (spinous  processes) 
o  Design  of  fluid  control  manifold  system  for  distribution  of  high  and  low  pressure  blood 
simulant,  low  pressure  clear  fluid  for  CSF,  aqueous  humor 
o  Design  of  fractured  jaw  and  maxilla/face  components  allowing  for  mobile  bone 
segments  to  be  palpated  under  mannequin  skin 
o  Mechanical  platform  combining  pitch/yaw/proptosis/blinking  action  for  establishing 
sense  of  "presence"  and  demonstrating  relevant  signs  of  injury 
o  Creation  of  unified  tarsal  plate/ligament  structures  for  improved  realism,  addition  of 
vessels  and  canaliculi  into  eyelid  models,  sensing  wires  into  canthal  ligaments  for 
detection  of  cantholysis 

•  Integration  of  head/neck  structures  with  commercial  mannequin  system 

o  Evaluated  multiple  commercial  mannequin  systems  for  ease  of  compatibility  with 
replacement  of  stock  head/neck  with  alternate  eye/face-trauma  specific  replacement 
o  Creation  of  software  architecture  to  link  mannequin  SDK  to  eye/face-trauma  specific 
controls  with  capability  for  scenario-specific  interventions  to  influence  behavior  of 
overall  mannequin  operation,  sample  physiological  parameters  set  by  instructor  tablet, 
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overwrite  physiological  parameters  from  simulator  software,  enable/disable  features  of 
the  mannequin  control  system 

o  Designed  &  fabricated  mounting  structure  for  neck  that  can  be  modified  easily  for 
compatibility  with  other  mannequin  types/vendors 
o  Evaluated  and  developed  air  muscle  actuation  of  animatronic  neck  including  range  of 
motion  based  on  mannequin  compressed  air 

o  Established  methods  for  linking  simulator  software  with  mannequin  software  to  sample 
physiological  parameters  set  by  instructor  tablet,  overwrite  physiological  parameters 
from  simulator  software,  enable/disable  features  of  the  mannequin  control  system 
o  Developed  microcontroller-based  interface  between  head/neck  controller  and 
solenoid/servo/sensor  elements  within  the  head/neck 

•  Conduct  data  collection  exercises/basic  validation  with  surgeons,  fellows,  residents 

o  Prepared,  amended  human  subjects  research  protocol  to  support  initial  and  ongoing 
data  collection  in  cooperation  with  annual  USUHS  Ocular  Trauma  course 
o  Collection  of  ~50  data  sets  for  lid  laceration  repair  from  attending  physicians,  fellows 
and  PGY2-PGY4  residents 

o  Preliminary  analysis  of  data  towards  identifying  important  geometric,  instrument  action 
thresholds 

o  Support  of  Simulator  Validation  Program  study  at  USUHS  comparing  simulator  training 
for  lid  laceration  repair  with  animal  tissue-based  prior  gold-standard 
o  Collection  of  survey  data  for  feedback  on  simulator,  results  showing  areas  for 
improvement  of  physical  structure,  anatomical  models  and  presentation  of  content 
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4.  Reportable  Outcomes 

Conference  slides,  abstracts,  posters  and  the  patent  application  documentation  are  available  on 
request. 

4.1.1.  Conference  papers  and  oral  presentations 

Ottensmeyer  MP,  Ahn  B,  Moore  JC,  Munteanu  E,  lorino  S,  De  Novi  G,  Bardsley  R,  Shah  R,  Neumann  P, 
Langston  D.  Ocular  and  Craniofacial  Trauma  Treatment  Training  System:  Overview  &  Eyelid  Laceration 
Module.  Workshop  Proceedings  of  the  8th  International  Conference  on  Intelligent  Environments.  J.A. 
Botia  et  al.  (Eds.)  IOS  Press,  2012.  pp  319-330. 

De  Novi  G,  Bardsley  R,  Shah  R,  Ottensmeyer  MP,  Moore  JC,  Ahn  B.  Event-driven  Surgical  Gesture 
Segmentation  and  Task  Recognition  for  Ocular  Trauma  Simulation.  Workshop  Proceedings  of  the  8th 
International  Conference  on  Intelligent  Environments.  J.A.  Botia  et  al.  (Eds.)  IOS  Press,  2012.  pp  341-352. 

4.1.2.  Conference  and  workshop  abstracts  and  posters 

Ottensmeyer  MP,  De  Novi  G,  Bardsley  RS,  Moore  JC,  Ahn  B,  lorino  S,  Munteanu  E,  Troulis  MJ,  Mazzoli 
RA,  Dawson  SL.  Ocular  and  Craniofacial  Trauma  Treatment  Training  System:  Lid  &  globe  laceration 
module  work  in  progress  &  program  overview  /  Poster  presentation,  Medicine  Meets  Virtual  Reality, 
Newport  Beach,  CA,  9-11  Feb  2012 

Ahn  B,  Moore  JC,  Munteanu  E,  lorino  S,  Bardsley  RS,  De  Novi  G,  Dawson  SL,  Troulis  MJ,  Ottensmeyer 
MP.  Ocular  and  Craniofacial  Trauma  Treatment  Training  System:  Trauma  module  animatronics  and 
structure,  simulator  interface  /  Poster  presentation,  65th  Annual  SAC  Poster  Session,  Boston,  MA, 18-19 
Apr  2012 

De  Novi  G,  Bardsley  RS,  Shah  R,  Ottensmeyer  MP.  Event-driven  Surgical  Gesture  Segmentation  and  Task 
Recognition  for  Ocular  Trauma  Simulation  /  Poster  presentation,  65th  Annual  MGH  SAC  Poster  Session, 
Boston,  MA,  18-19  Apr  2012 

Ahn  B,  Moore  JC,  Munteanu  E,  Ottensmeyer  MP.  Eye  Motion  and  Pupil  Contraction  Modular  Systems  for 
Eye  Trauma  Treatment  Training  Simulator  /  Poster  presentation,  MGH  ORCD  Poster  Celebration,  Boston, 
MA,  24  May  2012 

De  Novi  G,  Bardsley  RS,  Shah  R,  Ottensmeyer  MP.  Event-driven  Surgical  Gesture  Segmentation  and  Task 
Recognition  for  Ocular  Trauma  Simulation  /  Poster  presentation,  MGH  ORCD  Poster  Celebration,  Boston, 
MA,  24  May  2012 

Ottensmeyer  MP,  De  Novi  G,  Ahn  B,  Bardsley  RS,  Moore  JC,  Shah  R,  Munteanu  E,  lorino  S,  Dawson  SL. 
Development  of  an  Ocular-Facial  Trauma  Treatment  Training  Simulator  /  Poster  presentation,  Military 
Health  System  Research  Symposium,  Ft.  Lauderdale,  FL,  13-16  Aug  2012 

Ahn  B,  Moore  JC,  Zivieri  S,  De  Novi  G,  Bardsley  RS,  Ottensmeyer  MP.  Ocular  and  Craniofacial  Trauma 
Simulator  Development:  AR  Surgical  Microscope,  CT-Derived  Physical  Anatomy,  Sensor  &  Actuator 
Systems  /  Poster  presentation,  MGH  66th  Annual  SAC  Poster  Session,  Boston,  MA  20-21  Mar  2013 
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De  Novi  G,  Bardsley  RS,  Festi  M,  Ottensmeyer  MP.  Ocular  and  Craniofacial  Trauma  Simulator:  Event 
Driven  Gesture  Recognition,  Contextual  Feedback  Interface  and  User  Trial  Result  /  Poster  presentation, 
MGH  66th  Annual  SAC  Poster  Session,  Boston,  MA  20-21  Mar  2013 

Ottensmeyer  MP,  De  Novi  G,  Loan  G,  Pifferi  F,  Festi  M.  Development  of  an  Ocular-Facial  Trauma 
Treatment  Training  Simulator  /  Poster  presentation,  oral  presentation,  Military  Health  System  Research 
Symposium,  Ft.  Lauderdale,  FL,  11-15  Aug  2013 

Ottensmeyer  MP,  De  Novi  G.  Conversion  of  stereo  surgical  microscope  for  augmented  reality  application 
in  an  eye  trauma  simulator  /  Poster  presentation,  Medicine  Meets  Virtual  Reality,  Manhattan  Beach,  CA, 
19-22  Feb  2014 

De  Novi  G,  Loan  G,  Ottensmeyer  MP.  Ocular  and  Craniofacial  Trauma  Simulator:  Event  Driven  Gesture 
Recognition,  Contextual  Feedback  Interface  and  User  Trial  Result  /  Poster  presentation,  International 
Meeting  on  Simulation  in  Healthcare,  San  Francisco,  CA,  25-29  Jan  2014 

De  Novi  G,  Loan  GJ,  Gallerani  M,  Pifferi  F,  Ottensmeyer  MP.  Ocular  and  craniofacial  trauma  treatment 
training  simulator:  hardware  advances  and  user  feedback  /  Poster  presentation,  Military  Health  System 
Research  Symposium,  Ft.  Lauderdale,  FL,  18-24  August  2014 

De  Novi  G,  Loan  GJ,  Ottensmeyer  MP.  Progress  on  Ocular  and  Facial  Trauma  Training  Simulator  /  Poster 
presentation,  MGH  68th  Annual  SAC  Poster  Session,  Boston,  MA  1-2  Apr  2015 

4.1.3.  Invited  Presentations 

TATRC  display  area  demonstration  of  prototype  simulator  /  MMVR19,  Medicine  Meets  Virtual  Reality, 
Newport  Beach,  CA,  9-11  Feb  2012 

Eye  trauma  simulation  -  review  and  novel  system  development  /  Oral  presentation  /  5th  Military  Vision 
Symposium  on  Ocular  and  Vision  Injury,  Boston,  MA,  18  Sep  2012 

Simulation  -  grand  concepts  &  eye  trauma  example  /  Oral  presentation  /  Vision  Center  of  Excellence 
Roadmap  Meeting,  2012,  Boston,  MA,  17-19  Oct  2012 

Ottensmeyer  MP.  Ocular  Trauma  Training  Simulator:  Work  in  progress  -  system,  gesture  tracking,  user 
trial.  /  Oral  presentation  /  MMVR  2013,  San  Diego,  CA,  21  Feb  2013 

Ottensmeyer  MP,  De  Novi  G,  Bardsley  RS.  Ocular  and  Craniofacial  Trauma  Treatment  Training  System: 
Program  overview  &  work  in  progress  /  TATRC  display  area  demonstration  and  poster  /  MMVR20, 
Medicine  Meets  Virtual  Reality,  San  Diego,  CA,  21  Feb  2013 

4.1.4.  Student  Theses 

Master's  thesis,  Stefano  lorino,  Politecnico  De  Milano,  Facolta'  Del  Design,  Italy:  "TrackSim:  UN 
SIMULATORE  Dl  INTERVENE  CHIRURGICI  PER  TRAUMI  OCULARI  E  CRANIOFACCIAU"  ("TrackSim:  A 
simulator  for  eye  trauma  and  craniofacial  surgery"),  2012. 
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Master's  thesis,  Evgheni  Munteanu,  Universita'  Di  Bologna,  Facolta'  De  Ingegneria,  Italy:  "Progetto  di  un 
Sistema  de  elaborazione  di  segnali  biomedicali  per  applicazioni  di  chirurgia  ocular"  ("A  Biomedical  Signal 
Processing  System  for  Ocular  Surgery"),  2012. 

Master's  thesis,  Simone  Zivieri,  Universita'  Degli  Studi  Di  Modena  E  Reggio  Emilia,  Ingegneria  "Enzo 
Ferrari",  Italy:  "EYE-MECH:  Integrated  Design  of  a  Robotic  Eye  for  Application  in  Surgical  Simulators", 
2013. 

Master's  thesis,  Martina  Festi,  Universita'  Degli  Studi  De  Bologna,  Facolta'  De  Ingegneria,  Italy:  "EYE- 
TRAUMA  Project:  Event  Driven  Control  Software  for  Gesture  Recognition  in  Medical  Simulation",  2013. 

Master's  thesis,  Federico  Pifferi,  Universita'  Degli  Studi  Di  Modena  E  Reggio  Emilia,  Ingegneria  "Enzo 
Ferrari",  Italy:  "Virtual  and  Physical  Prototyping  of  the  Neck  of  a  Humanoid  Robot  for  Surgical 
Simulators",  2013. 

Master's  thesis,  Marcello  Gallerani,  Universita'  Degli  Studi  Di  Modena  E  Reggio  Emilia,  Ingegneria  "Enzo 
Ferrari",  Italy:  "Integrated  Design  and  Prototyping  of  a  Robotic  Head  for  Ocular  and  Craniofacial  Trauma 
Simulators",  2014. 

4.1.5.  Invention  disclosures  and  patent  applications 

MGH  Invention  Disclosure,  "Ocular  Trauma  Treatment  Training  Systems".  Case  number  22084,  logged 
02/25/2013. 

MGH  Invention  Disclosure,  "Training  Simulator  for  Ocular  Trauma  and  Other  Domains  Using  Event-Based 
Surgical  Gesture  Detection  and  Multiple  Instrument  and  Hand  Motion  Tracking  Systems".  Case  number 
22275,  logged  06/11/2013. 

4.1.6.  Patent  Applications 

United  States  Provisional  Patent  Application,  "Augmented  Reality  Microscope".  Serial  number 
61/766,987,  filed  02/20/2013,  serial  number  61/860,397,  filed  07/31/2013,  serial  number  61/861,179, 
filed  08/01/2013. 

United  States  Patent  Application,  "Trauma  Training  Simulator  With  Event-Based  Gesture  Detection  and 
Instrument-Motion  Tracking".  Ottensmeyer  MP,  De  Novi  G,  Loan  GJ,  Bardsley  RS,  Moore  JC.  Serial 
number  20150037775,  filed  06/30/2014. 

4.1.7.  Licenses 

Negotiations  ongoing  towards  option  to  license  technology  by  Simulation  Systems,  LLC  pending  further 
progress  on  STTR  proposal  submission. 
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4.1.8.  Training  experiences: 

Post-doctoral  research  fellowship:  Gianluca  De  Novi,  Ph.D.  (U.  Bologna)  Robotics.  Developing  software 
architecture  and  implementation  of  simulator  interface.  Hired  as  permanent  employee,  currently  Lead 
Software  Architect  for  MGH  Simulation  Group. 

Post-doctoral  research  fellowship:  Bummo  Ahn,  Ph.D.  (KAIST,  South  Korea)  Mechanical  engineering. 
Developing  mechatronic  systems  for  simulator,  instrument  motion  sensing,  intervention  sensing. 
Returned  to  South  Korea,  currently  Senior  Researcher,  Korea  Institute  of  Industrial  Technology. 


Internship:  Stefano  lorino,  master's  candidate,  Design  and  Engineering,  Faculty  of  Design  Politecnico  di 
Milano,  Milan,  Italy.  6-month  term  working  on  system  structural  design,  instrument  connection  system, 
lighting  system,  AR  microscope  prototype  optics. 

Internship:  Evgheni  Munteanu,  master's  candidate,  Mechatronics,  Department  of  Electronics 
Informatics  and  Systems,  Faculty  of  Engineering,  University  of  Bologna,  Bologna,  Italy.  4-month  term 
working  on  developing  USB-based  interfaces  for  surgical  instrument  motion  tracking,  control  of 
simulator  lighting  system. 

Internship:  Simone  Zivieri,  master's  candidate,  mechanical  engineering,  University  of  Modena  and 
Reggio  Emilia.  6-month  work  term  learning  mechanical  design,  CAD,  control  of  servomechanisms, 
calibrating  elastic  response  of  transmission;  working  on  the  eye  motion  mechatronics.  Graduated  cum 
laude. 

Internship:  Martina  Festi,  master's  candidate,  computer  science,  University  of  Bologna.  6-month  work 
term  learning  to  use  C#  libraries,  bridging  methods  between  C#  and  C++,  as  applied  to  interfacing 
simulator  to  Laerdal  mannequin  SDK,  assembly  language  coding  as  applied  to  simulator  data  acquisition 
microcontroller.  Graduation  expected  towards  end  of  2013. 

Internship:  Federico  Pifferi,  master's  candidate,  mechanical  engineering,  University  of  Modena  and 
Regio  Emilia,  6-month  work  term  learning  CAD,  mechanical  design,  control  of  pneumatic  actuators, 
design  of  pneumatic  actuators.  Working  on  design  of  animatronic  neck  of  mannequin. 

Internship:  Marcello  Gallerani,  master's  candidate,  mechanical  engineering,  University  of  Modena  and 
Regio  Emilia,  6-month  work  term,  learning  CAD,  mechanical  design,  control  of  servos,  stepper  motors. 
Working  on  design  of  animatronic  jaw,  head  motion  of  mannequin. 

4.1.9.  Funding  applications 

Participated  in  preparation  of  pre-proposal  and  invited  full  proposal  in  response  to  W81XWH-12-JPC1- 
DTME-PRP,  "Developer  Tools  for  Medical  Education  Public  Physiology  Research  Platform".  Proposal 
entitled  ""iBody":  a  Unified  Public  Physiology  Research  Engine  -  Open  Infrastructure  Enabling  an 
Integrated  Continuum  of  Virtual  and  Physical  Simulation  Technology  Platforms"  under  co-PIs  James  A. 
Gordon,  M.D.  and  John  Parrish,  M.D. 
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Selected  as  subawardee/contractor  for  "Feasibility  and  Implementation  of  Ocular  Injury  Simulator 
Surgical  Systems  in  Training  Military  Ophthalmologists",  proposal  for  Medical  Simulator  Validation 
Program  intramural  solicitation  (SVP),  Marcus  Colyer,  M.D.,  P.l.  Provided  access  to  simulator  and 
production  of  consumable  trauma  modules,  materials  and  travel  expenses  to  be  reimbursed  for  2013, 
2014  and  2015  USUHS  Ocular  Trauma  Course  events. 

Proposal  awarded:  "Eye/Face  Trauma  Simulation  Enhancements",  Mark  P.  Ottensmeyer,  Ph.D.,  P.l. 
Subaward  of  USAMRMC  W81XWFI-14-2-0008  (Joint  Warfighter  Medical  Research  Program),  extending 
work  previously  supported  by  CIMIT.  Proposal  to  develop  mid-facial  fracture  module,  adapt  simulator 
to  be  compatible  with  CIMIT  modular  mannequin  system  under  development. 

Proposal  awarded:  "Advanced  Modular  Manikin",  James  A.  Gordon,  M.D.,  P.l.  USAMRMC  W81XWH-14- 
C-0100.  Program  to  develop  platform  technology  for  a  modular  manikin  systems,  adapting  ocular- 
craniofacial  simulator  for  use  as  one  possible  head  module  for  this  system. 

Work  in  progress  on  NIH  PA-15-270  STTR  proposal  with  Simulation  Systems,  LLC.  Planned  research  and 
development  will  address  verification  and  validation  of  simulator  data  collection  and  measurement  of 
trainee  performance  vs.  expert  performance. 
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5.  Conclusion 

This  report  presents  the  full  scope  of  the  progress  made  to  create  a  novel  eye  and  face  trauma  simulator 
for  use  by  trainee  surgeons  and  those  new  or  returning  to  eye  trauma  practice  as  well  as  first 
responders  facing  eye  trauma  and  likely  coincident  facial  trauma  in  the  field. 

The  system  that  we  developed  is  a  prototype  that  presents  a  solution  to  a  variety  of  gaps  in  the  available 
training  methods  in  this  domain. 

Firstly,  at  the  outset  of  the  program,  there  were  no  available  commercial  systems  that  provided 
simulated  training  experiences  in  responding  to  ocular/periocular  eye  trauma.  There  were  VR-based 
systems  that  focused  on  procedures  such  as  cataract  and  retinal  surgery  and  simple  part  task  trainers  for 
globe  surgery  (the  above,  plus  strabismus,  enucleation,  etc.),  but  no  simulation-based  systems  for 
eye/periocular  trauma  and  oculoplastics.  With  the  drive  towards  eliminating  animal-based  training  and 
the  challenge  of  learning  procedures  on  patients  in  need  of  surgery  done  correctly  the  first  time,  the 
need  for  a  simulator  to  fill  this  gap  was  clear  and  this  system  provides  such  a  solution. 

Next,  there  is  a  growing  need  for  quantitative  evaluation  capabilities  in  training  to  verify  that  a  trainee 
has  met  meaningful  criteria  for  successful  learning,  and  these  criteria  should  be  more  fine-grained  than 
a  global  score  based  on  gross  features  such  as  overall  path  length  or  elapsed  time.  Identifying  specific 
areas  for  improvement  can  help  to  focus  training  so  that  it  can  be  made  more  efficient,  which  is 
important  in  the  era  of  limited  resident  hours  and  growing  recognition  of  the  costs  of  training  in  an  OR 
environment.  To  further  enhance  the  learning  experience,  having  an  automated  system  that  captures 
the  experience  of  the  senior  physician  "looking  over  the  shoulder"  of  the  trainee  to  provide  context¬ 
relevant  corrections  and  additional  teaching  material  would  be  an  improvement  over  inanimate  system 
and  would  leverage  the  skills  of  a  single  expert  across  multiple  trainees.  Again,  this  simulator  presents  a 
solution  that  approaches  these  challenges. 

Lastly,  our  original  motivation  was  to  address  needs  that  span  the  spectrum  of  care  providers.  In 
particular  we  aimed  to  create  something  that  would  support  the  needs  of  the  first  responders,  who  had, 
at  the  outset,  very  few  options  to  provide  a  realistic  experience  of  eye  trauma,  compared  with  the 
relatively  robust  general  medical  simulator  marketplace.  At  the  same  time,  we  aimed  to  create  a  system 
that  would  enable  transmission  and  preservation  of  the  skills  of  physicians  with  combat-trauma 
response  skills  to  the  next  generations  of  physicians,  in  an  environment  where  certain  basic  skills  were 
being  deemphasized  in  civilian  training  and/or  are  increasingly  limited  to  a  smaller  number  of  regional 
centers.  Creating  a  system  that  included  components  that  could  be  integrated  both  with  a 
benchtop/"wet-lab"  simulator  and  with  full-body  mannequins  was  a  challenge  that  attempted  to  reach 
both  audiences  with  relevant  features  while  reusing/repurposing  as  many  of  the  common  features  as 
possible. 


As  distilled  in  the  Key  Research  Accomplishments,  we  were  successful  in  constructing  a  system  that 
provides  capabilities  addressing  each  of  the  gaps.  The  final  CT-derived  anatomical  features  provided  a 
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diversity  of  options  to  provide  training  experiences  for  both  the  surgeons  and  the  first  responders, 
through  presentation  of  lacerations  through  multiple  tissue  layers  for  repair  or  palpable  indications  of 
facial  fracture  or  integrated  moulage  prompting  responses  such  as  cricothyroidotomy.  The  suite  of 
sensors,  distributed  across  surgical  instruments  and  the  anatomical  components  themselves  (e.g.  force 
sensing  eye  globe  supports,  cut  sensors  for  detecting  cricothyroidotomy  incision  length  and  orientation) 
similarly  allow  detection  both  of  correct  but  also  incorrect  response  to  the  given  scenario,  and  helps  to 
avoid  "gaming"  the  system.  The  animatronic  components  of  eye/lid/head  motion  and  features  intended 
to  generate  blood  simulant/fluid  flow  support  increased  realism  to  help  to  immerse  the  trainee  in  the 
scenario  and  reinforce  the  learning  experience.  The  software  architecture  integrates  the  sensing 
capabilities,  3D  model  based  information  about  the  locations  of  instruments  and  sensors  and  knowledge 
derived  from  experts  to  identify  how  a  trainee  is  performing  an  intervention  and  present  context¬ 
relevant  information  to  enhance  the  learning  experience. 

The  system  represents  a  platform  technology  that  is  going  on  to  support  further  developments  in  the 
ocular/facial  trauma  domain.  We  were  part  of  the  Massachusetts  General  Hospital  team  that  pursued 
one  of  the  phase  1  Advanced  Modular  Manikin  research  and  development  efforts,  in  which  we  adapted 
and  expanded  upon  the  head/neck  structure  and  the  electronic  interface  to  make  it  compatible  with  our 
designs  and  prototype  of  the  AMM.  The  full-face  modules  are  under  consideration  as  part-task  trainers 
on  their  own  to  support  lower  cost  training  or  skills  refreshing  for  canalicular  stenting  and  lid/globe 
laceration  repair  (omitting  the  sensing/software  elements).  The  Event-Driven  Surgical  Gesture 
Recognition  approach  is  a  generic  tool  that  can  be  applied  just  as  well  to  other  domains  of  instrument- 
mediated  medicine  (minimally  invasive/robotic  surgery,  microsurgery,  etc.),  and  we  are  exploring  where 
we  will  be  able  to  apply  the  technique  in  future  work.  We  are  investigating  further  funding 
opportunities  with  a  commercial  partner  to  expand  the  verification/validation  work  in  support  of  a 
transition  to  a  commercial  version  of  the  system.  And  lastly,  we  were  successful  in  applying  for  funding 
to  enhance  the  system  with  additional  features  including  a  more  advance  surgical  fracture  repair  feature 
set  (i.e.  stabilizing  the  bone  fragments  with  plates  and  screws)  and  an  enucleation  module  for  injuries  so 
severe  that  salvage  of  the  globe  would  not  be  possible. 

Perhaps  most  importantly,  we  were  successful  in  creating  a  prototype  that  had  utility  for  training  not 
just  after  completion  but  also  in  the  course  of  development.  We  are  incredibly  grateful  for  the  support 
and  expertise  of  Drs.  Richard  Stutzman  and  Marcus  Colyer,  course  directors  of  the  USUHS  Ocular 
Trauma  Course,  and  the  opportunity  to  provide  the  system  for  use  in  training  by  the  course  faculty  for 
the  students,  so  that  the  students  had  the  opportunity  to  learn  techniques  on  simulated  human 
anatomy  (vs.  animal).  Because  of  the  unplanned,  extended  period  of  performance,  we  had  the 
opportunity  to  observe  and  learn  from  the  interactions  of  the  faculty  and  students  with  the  system  and 
make  more  improvements  to  it  than  might  have  been  possible  under  the  original  calendar. 


The  last  of  the  preceding  successes  hints  at  some  of  the  areas  in  which  this  research  effort  fell  short  of 
measuring  up  to  our  original  aspirations.  The  commencement  of  the  program  came  in  the  shadow  of  a 
funding  gap  that  saw  most  of  the  originally  proposed  team  temporarily  leaving  MGH,  resulting  in  a 
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slower  than  desired  start  to  the  program  as  original  team  members  returned  in  some  cases  from 
geographic  relocation  and  new  personnel  were  recruited,  and  a  lag  in  the  calendar.  That  lag  led  to 
mismatch  between  some  of  our  early  goals  for  user  testing  and  the  best  scheduled  opportunities  to 
conduct  that  testing.  With  the  benefit  of  hindsight,  the  linear,  sequential  structure  of  the  milestones 
that  was  proposed  was  at  odds  with  the  nature  of  the  research  effort,  which  is  rarely  a  linear  effort  as 
new  technologies  and  ideas  emerge  that  could  not  be  predicted  in  the  original  proposal.  We  greatly 
appreciate  the  flexibility  that  was  shown  by  our  sponsors  in  granting  no-cost  extensions  that  helped  to 
accommodate  the  divergence  between  plan  and  execution  and  enabled  the  aforementioned  additional 
opportunities  to  work  with  the  end  user  faculty  members  and  students. 

This  project  also  illustrated  the  importance  of  excellent  project  planning,  management  and  tracking 
skills  as  part  of  a  team  in  parallel  with  the  R&D  expertise.  With  respect  to  the  emphases  of  the  project, 
we  were  very  successful  in  addressing  the  slice  of  the  simulator  that  responded  to  the  training  needs  for 
a  lid  laceration  scenario,  but  less  so  regarding  the  other  surgical  tasks  (canalicular  repair,  retrobulbar 
hemorrhage  and  cricothyroidotomy)  and  the  manikin  integration  element  -  physical  components  were 
developed  that  support  additional  work  towards  complete  scenarios,  but  did  not  reach  the  same  level  as 
that  of  the  lid  laceration.  The  need  to  maintain  balance  between  all  of  the  elements  of  the  plan  is  one 
of  the  lessons  learned  for  future  work.  Also  related  to  the  management  element,  the  PI  had  significant 
challenges  in  meeting  the  reporting  requirements  of  the  program,  initially  as  a  result  of  the  happy 
addition  of  twin  infants  during  Year  3,  but  later  as  a  result  of  over-commitment  across  a  number  of 
research  programs;  proposals  for  other  research  that  were  prepared  since  this  program  began 
specifically  called  out  for  additional  personnel  to  aid  in  meeting  such  requirements  in  the  future. 


In  the  third  PMI  category  of  (Plus,  Minus)  and  Interesting,  as  a  primarily  research  oriented  effort,  the 
simulator  that  was  developed  incorporated  a  much  higher  density  of  features  that  would  likely  be 
desirable  in  a  commercial  product.  Combining  the  position  tracking,  AR  projection,  AR  microscope,  dual 
use  capability  of  the  head/neck  form  as  both  the  bench-top  and  mannequin-integrated,  etc.  could  result 
in  a  product  that  would  be  overly  complex  and  expensive.  That  point  is  well  taken;  the  direction 
towards  which  we  expect  the  system  will  evolve  would  be  to  select  the  most  relevant  subset  of  features 
for  particular  applications,  and  with  the  benefit  of  the  modularity  that  we  built  into  the  prototype, 
relatively  quickly  adapt  the  design  and  capabilities  of  a  new  system  for  those  more  specific  applications. 
We  followed  this  path  in  the  adaptation  of  the  head/neck  design  for  compatibility  with  the  AMM 
project,  which  retained  the  animatronic  and  anatomical  features,  while  dispensing  with  the  instrument 
tracking/gesture  recognition  elements.  Such  down-selection  restores  the  focus  that  will  be  necessary 
for  commercial  successful  spin-off  elements  of  this  project. 


In  terms  of  overall  impact  of  this  program,  it  is  already  clear  that  the  simulator  will  not  be  orphaned 
following  submission  of  this  final  report.  As  described,  the  results  obtained  to  date  have  supported  a 
successful  grant  proposal  to  extend  the  capabilities  of  the  system,  which  will  open  additional 
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opportunities:  the  enucleation  component  under  development  bears  similarity  to  the  requirements  for 
strabismus  surgery,  which  is  of  interest  to  physicians  at  Boston  Children's  Hospital,  with  whom  we  have 
had  preliminary  discussions.  The  intellectual  property  developed  was  deemed  by  our  institution  to  be  of 
sufficient  value  to  warrant  conversion  of  the  patent  application,  with  the  expectation  that  the  costs  will 
be  recouped  through  future  licensing  opportunities.  And  last,  but  not  lease,  in  addition  to  the  series  of 
data  collection  and  Simulator  Validation  Program  efforts  in  which  we  participated  during  the  extended 
period  of  performance,  the  system  continued/continues  to  be  used  at  the  USUHS  Ocular  Trauma 
courses  in  2015  and  planned  for  2016. 
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7.  Appendices 


7.1.  List  of  all  Personnel  Receiving  Pay 

Mark  P.  Ottensmeyer,  Ph.D.  (PI) 

Steven  L.  Dawson,  M.D.  (SME) 

Maria  J.  Troulis,  D.D.S.  (SME) 

Ryan  S.  Bardsley  (systems  designer) 

John  Cho  Moore  (industrial  designer) 

Gianluca  De  Novi,  Ph.D.  (software  architect) 

Gregory  J.  Loan  (clinical  engineer) 

Bummo  Ahn  (post-doctoral  research  fellow) 

Rajesh  Shah  (biomedical  engineer) 

David  Brennan  (imaging  technician) 

Daniel  Langston  (contractor) 
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Abstract.  Eye  trauma  in  combat-related  injuries  can  be  devastating.  Body  armor 
improvements  increase  survivability  from  many  injuries,  however  the  face  remains 
relatively  unprotected,  especially  from  IED  blast  and  fragments.  The  number  of 
physicians  with  significant  eye  trauma  expertise  is  limited  due  to  lack  of  teaching 
opportunities  and  evolution  of  surgical  techniques  away  from  micro  suturing.  To 
provide  a  non-animal,  non-cadaver  training  system,  a  hybrid  physical/virtual  eye 
trauma  simulation  system  for  surgical  technique  learning  and  evaluation  is  being 
developed.  It  includes  a  silicone/polymer  head  with  replaceable  eye  trauma 
modules  with  realistic  anatomy.  Position/orientation  and  grasp  force/position  of 
surgical  instruments  are  measured,  and  real  time  scoring  and  feedback  methods  are 
under  development.  These  elements  are  integrated  with  teaching  materials 
presented  on-screen  and  during  training  scenarios  via  augmented  reality  projection. 
The  initial  efforts  focus  on  a  lacerated  eye  lid  scenario.  Developments  towards 
additional  trauma  modules  with  animatronic  features  are  discussed. 

Keywords.  Eye  trauma,  surgical  gesture  measurement,  augmented  reality 


1.  Introduction 

1.1.  Motivation 

Craniofacial  and  ocular  injuries  have  been  elements  of  up  to  29%  of  all  battlefield 
injuries  suffered  by  US,  UK  and  allied  forces  in  recent  conflicts  in  Iraq  and 
Afghanistan.  While  improvements  in  armor  systems  have  reduced  fatality  due  to 
injuries  to  the  torso,  the  face  and  head  remain  relatively  vulnerable  and  casualties 
present  with  more  injuries  to  the  eyes  and  face  than  proportional  relative  to  body 
surface  area.  Emphasis  on  the  use  of  eye  protection  reduced  the  prevalence  of  eye 
injuries  from  Operation  Desert  Storm  in  1991  (13%)  to  Operations  Iraqi  Freedom  and 
Enduring  Freedom  in  2001-2005  (6%)[1].  Strict  compliance  with  eye  protection  usage 
can  reduce  eye  injuries  to  0.5%  in  some  cases  [2],  however  this  is  typically  not 
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achieved,  and  rates  vary  [3].  The  eye  is  sensitive  to  blast  injuries  and  without  proper 
eye  protection,  many  small  fragments  that  might  not  penetrate  skin,  are  likely  to  do 
damage  to  the  eye  and  surrounding  tissues. 

At  the  same  time  as  soldiers  are  surviving  formerly  fatal  injuries,  but  with 
disfiguring  and  disabling  eye/face  wounds,  the  skills  required  to  address  them  are  not 
as  broadly  available  as  they  once  were.  Microsuturing  has  been  superseded  by 
sutureless  methods  for  certain  procedures,  so  the  relevant  skills  are  not  as  frequently 
used.  Trauma  cases  are  typically  referred  to  trauma  centers,  so  fewer  ophthalmologists 
have  exposure  to  relevant  cases,  and  the  uniqueness  of  each  trauma  case  makes  the 
learning  process  a  lengthy  one.  In  addition,  there  are  financial  disincentives  to 
specialize  in  trauma  vs.  other  ophthalmology  specialties,  involving  reimbursement 
levels  for  re-operations  for  a  given  case. 

Alternative  training  modes  include  use  of  cadaver  tissues  [4],  animal  eyes  and 
tissues  [5],  and  simulated  anatomy  [6].  Cadaver  and  animal  eye  tissue  typically  has 
different  mechanical  behavior  from  live  tissue.  Further,  there  is  a  strong  push  for 
military  medical  training  to  reduce  or  eliminate  the  use  of  tissue  for  training  purposes. 
With  both  tissue  training  and  the  use  of  simulated  anatomy,  there  is  a  need  for  senior 
oversight  and  guidance,  which  is  an  expensive  use  of  a  skilled  surgeon’s  time  which 
needs  to  be  balanced  against  time  spent  in  the  OR. 

There  is  a  need,  therefore,  to  provide  more  and  better  opportunities  for  self- 
directed,  repeated  and  repeatable,  diverse  and  relevant  synthetic  training  options.  In 
support  of  this,  we  are  developing  an  eye  and  facial  trauma  simulation  system, 
integrating  not  only  a  physical  mannequin,  but  the  enhancements  provided  by  multiple 
modes  of  content  and  guidance  presentation,  multiple  modes  of  sensing,  and  the 
capability  for  presenting  a  variety  of  trauma  cases  using  the  same  base  platform. 

1.2.  Previous  Research 

Existing  eye  surgery  simulation  systems  tend  towards  opposite  ends  of  the  virtual  vs. 
fully-physical  spectrum.  In  the  virtual  space,  there  are  a  variety  of  simulators  focused 
on  non-trauma  scenarios,  including  cataract  surgery  [7]  [8].  A  recent  entrant  to  the 
virtual  arena  is  [9].  Primary  drawbacks  of  fully  virtual  systems  is  that  they  either 
dispense  with  tactile  and  force  feedback  components,  in  a  domain  where  careful  control 
of  applied  forces  is  crucial,  or  when  force  feedback  is  included,  it  relies  on  typical 
commercial  haptic  interfaces,  which  are  difficult  to  integrate  within  the  tight  confines 
around  the  eye.  Further,  multiple  instruments  are  used  during  the  course  of  a  typical 
procedure  (multiple  forceps,  needle  holders,  scissors,  accessories  such  as  surgical 
sponges),  which  must  be  “faked”  if  a  common  interface  handle  is  used,  or  detachable 
connectors  designed  to  allow  swapping  of  instruments. 

In  the  domain  of  the  fully  physical  are  a  number  of  commercial  products  [6]  [10]. 
These  have  varying  levels  of  accuracy,  typically  matched  closely  to  the  needs  for 
training  particular  procedures.  With  careful  choice  of  materials  used  in  fabrication,  they 
can  provide  accurate  tissue  behavior,  however  the  automatic  performance  measurement 
possibilities  of  the  virtual  are  unavailable.  Performance  evaluation  depends  on  the 
oversight  of  proctors  and  senior  physicians,  and  when  used  in  a  self-directed  fashion, 
there  would  be  minimal  opportunity  for  immediate  feedback  or  advice  on  improvement. 

One  notable  effort  to  combine  approaches  is  that  of  [11],  in  which  an  ex  vivo 
human  or  animal  eye  was  supported  in  a  cup  mounted  on  a  force  sensor,  to  evaluate 
loads  applied  to  the  eye.  Sensing  of  instrument  motions  was  not  performed. 
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The  general  focus  of  our  research  group  is  to  combine  the  most  useful  features  of 
the  physical  with  the  complementary  features  of  the  virtual  when  developing  medical 
training  simulators:  physical  realism,  so  force  feedback  is  obtained  passively  from 
physical  objects,  combined  with  tracking  and  responsiveness  capabilities  of  the  virtual 
-  measurement  of  instrument  trajectories,  sensing  of  interventions  and  automatic 
provision  of  real-time  feedback,  evaluation  and  guidance. 

One  example  of  this  work  was  a  collapsed  lung  response  simulator  [12]  [13],  which 
tracked  trajectories  of  “chest  tubes”  and  “chest  darts”,  causing  the  physical  mannequin 
to  respond  appropriately  to  insertion,  automatically  detecting  success  or  failure  and 
providing  immediate  feedback  to  the  trainee.  Similarly,  we  developed  an  early 
laparoscopy  interface  which  quantitatively  measured  physical  instrument  motions 
while  performing  basic  training  exercises,  yielding  performance  scores  relative  to  an 
aggregate  of  expert  performance  results  [14]  and  most  recently,  a  full-body  trauma 
victim  mannequin  for  use  in  training  for  severe  extremity  hemorrhage,  which  housed 
an  autonomous  control  unit  inside  the  body  of  a  physical  mannequin  [15]. 

1.3.  Eye  lid  anatomy 

The  system  currently  focuses  on  learning  techniques  relevant  to  the  repair  of  eye  lid 
lacerations.  The  eye  lid  (Figure  1)  is  made  up  of  the  layers  of  skin,  the  orbicularis  oculi 
muscle,  the  orbital  septum  (not  shown),  tarsal  plate  and  the  palpebral  conjunctiva.  The 
muscle  layer  extends  close  to  the  margin  of  the  lid  and  is  a  visible  dividing  line 
between  the  inner  and  outer  layers  of  the  lid,  known  as  the  grey  line.  Towards  the  inner 
comer  of  the  eye  (medial  canthus)  are  the  lacrimal  ducts,  which  drain  tears  away  from 
the  eye  and  into  the  nasal  sinus.  Additional  features  are  the  eyelashes,  growing  out  in 
front  of  the  grey  line,  and  the  openings  of  the  meibomian  glands,  behind  the  grey  line. 

For  the  purposes  of  the  initial  simulation,  we  consider  lacerations  through  the 
margin  of  the  lid,  in  regions  away  from  the  lacrimal  ducts,  so  the  users  will  be 
primarily  be  concerned  with  alignment  and  approximation  of  the  tarsal  plate,  skin 
surface  and  the  margin  of  the  eyelid. 

In  future  versions,  we  will  be  including  lacrimal  ducts,  which  require  more 
advanced  techniques  to  repair,  and  the  lateral  canthal  tendons,  which  are  dissected  as 
part  of  a  procedure  to  relieve  pressure  from  bleeding  in  the  eye  socket  often  due  to 
blunt  trauma.  Similarly,  while  the  current  globe  is  in  a  fixed  position,  future  versions 
will  have  both  passive  and  active  mobility,  as  described  below. 

1.4.  Eye  globe  anatomy 

Future  versions  of  the  simulator  will  focus  on  repair  of  injuries  to  the  anterior  anatomy 
of  the  eye  globe,  primarily  partial  and  full-thickness  lacerations  through  the  cornea  and 
sclera  (white).  The  major  stmctures  of  the  anterior  hemisphere  of  the  eye  (Figure  1) 
are  the  clear  cornea,  the  anterior  chamber,  filled  with  watery  aqueous  humor,  the  iris, 
which  contracts  and  dilates  under  the  control  of  the  ciliary  muscle,  the  lens  and  capsule, 
the  sclera  (white),  which  is  the  structural  wall  of  the  eye,  the  choroid  layer  inside  the 
sclera,  and  taking  up  most  of  the  volume  of  the  eye,  the  jelly-like  vitreous  humor. 
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Figure  1:  (left)  External  eye  showing  lacrimal  ducts,  (right)  Cross  section  of  the  eye  lid  and  globe.  Adapted 
from  http://en.wikipedia.0rg/wiki/File:Gray888.png,  http://en.wikipedia.Org/wiki/File:Gray896.png, 
http  ://en.  wikipedia.  org/wiki/F  ile:  Gray 1 205  .png 


2.  System  Overview 

Out  system  is  a  stand-alone  eye/face  trauma  simulator  with  augmented  reality 
teaching/evaluation  content.  It  consists  of  a  mannequin  head/neck  with  a  replaceable 
eyelid  and  eye  globe  module.  Titanium  ophthalmic  surgical  instruments  are  tracked  in 
six  degrees  of  freedom  (DOF)  and  grasping  action  and  grasping  force  are  measured.  In 
the  current  version,  the  AR  (augmented  reality)  hardware  is  comprised  of  a  Microsoft 
Kinect  and  a  pico-projector.  A  lighting  system,  stand,  tray,  electronics  for  the  surgical 
instruments,  and  a  host  PC  computer  complete  the  system  (Figure  2). 

The  physical  components  of  the  system  are  integrated  as  part  of  a  learning  system, 
which  will  keep  track  of  user  identity  and  performance,  present  teaching  content 
relevant  to  particular  training  scenarios,  monitor  performance  of  surgical  exercises 
during  the  particular  scenario,  and  present  feedback  during  and  after  the  scenario, 
including  performance  scores. 


Figure  2:  Stand-alone  ocular  and  craniofacial  trauma  treatment  training  system:  (left)  Mannequin  head/neck 
with  eyelid  portal  and  eye  globe,  and  hardware  architecture  for  augmented  reality  (AR)  (right)  Surgical 
instruments,  hot-swappable  connectors  and  signal  conditioning  circuitry. 
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2.1.  Anatomical  Modeling  and  Fabrication 

At  present,  the  major  anatomical  components  of  the  system  are  the  mannequin 
head/neck,  multiple  copies  of  a  module  including  the  tissue  of  the  eyelids,  brow,  and 
upper  cheek,  and  the  eye  globe  itself.  We  will  briefly  describe  the  fabrication 
techniques  for  each  of  these. 

2.1.1.  Head  form  -  skin  and  core 

With  the  participation  of  a  volunteer,  adult  male,  we  created  a  life  cast  (Figure  3)  using 
standard  techniques  and  materials  (skin  safe  silicone,  plaster  wrap)  to  create  the 
external  impression.  This  mold  was  removed  from  the  model  and  used  to  cast  a  clay 
first  copy  of  the  model’s  head.  This  clay  positive  was  then  textured  and  retouched  by 
hand  and  molded  again  to  create  the  final  mold  for  the  head.  A  plastic  positive  was  cast 
from  the  mold  and  laser  scanned,  so  that  a  graphical/CAD  model  with  identical 
dimensions  would  be  available  for  the  simulation.  A  second  positive,  made  of  molding 
clay,  was  shaved  down  to  approximate  the  shape  of  the  underlying  skull  based  on 
standard  thicknesses  of  soft  tissue  over  the  skull.  In  the  eye  region,  a  cavity  was  created, 
offsetting  the  surface,  so  that  replaceable  modules  fitting  within  this  space  could  be 
subsequently  fabricated.  Finally,  the  external  impression  and  the  internal  core  were 
reassembled,  with  an  additional  insert  occupying  the  space  of  the  eye  portal.  A  silicone 
skin,  with  the  correct  external  and  internal  contours  and  eye  portal  space  was  cast.  The 
CAD  model  was  used  to  create  a  rapid  prototyped  core  for  use  in  mounting  the 


Figure  3:  Molding  sequence  for  prototype  head-form.  Life  casting.  Clay  duplicate.  Plastic  positive  with  3D 
scanning  fiducial  marks.  Silicone  production  mold.  Plaster  core  with  3D  scanning  fiducial  marks,  air  holes 
for  skin  casting.  Final  molded  silicone  skin,  with  pigmentation  and  replaceable  trauma  modules. 


2.1.2.  Ocular  adnexa 

The  opening  in  the  skin  layer  surrounds  the  trauma  module  inserts,  which  include  the 
eyelids  and  surrounding  tissues,  both  cast  in  silicone.  The  eyelids  are  complex 
structures  that  include  multiple  tissue  layers  and  a  variety  of  complex  external 
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anatomical  features,  however,  most  significant  for  our  first  trauma  scenario  are  the  skin 
and  a  stiffer,  underlying  structure  called  the  tarsal  plate  (Figure  4). 

The  tarsal  plate  was  designed  by  first  laser  scanning  the  positive  plastic  casting  of 
the  eye  portal.  The  scanned  3D  model  was  then  imported  into  SolidWorks  CAD 
software  and  the  surface  model  of  the  eyelid  was  thickened  to  create  the  mold  for  the 
tarsal  plate.  The  mold  was  then  printed  on  an  Objet  Eden  260V  3D  printer  (Objet  Ltd., 
USA),  capable  of  printing  smooth  and  clear  surfaced  parts.  During  fabrication,  a  piece 
of  nylon  mesh  is  sandwiched  between  two  tarsal  plate  mold  parts  and  silicone  is 
injected  into  the  mold.  The  mesh  helps  to  limit  the  tendency  for  punctures  and  cuts  in 
the  silicone  to  propagate.  The  part  is  removed  from  the  mold  and  placed  in  the  portal 
mold  where  silicone  is  again  injected  to  create  the  eyelid  with  tarsal  plate. 


1 

1 

V  - 

k*  J 

_ f 

Figure  4:  Molding  of  eye  portal  -  production  mold,  early  single  material  eyelid  models;  eye  lids  with  "tarsal 
plates"  (back  and  front  after  suturing  during  data  collection  scenario). 


2.1.3.  Eye  globe 

The  current  version  of  the  globe  (Figure  5)  is  similarly  fabricated  in  various  grades  of 
silicone.  A  nylon  mesh  is  wrapped  around  a  two-piece  inner  mold  core.  An  external 
core  is  aligned  and  attached  to  the  inner,  and  white  pigmented  silicone  is  injected, 
forming  the  sclera  (white  of  the  eye).  A  stem  at  the  back  of  the  eye  allows  for  removal 
of  the  inner  core.  Following  removal,  a  molded,  clear  silicone  lens,  pigmented  iris  and 
clear  cornea  are  bonded  in  place  in  the  anterior  segment  of  the  eye  globe,  and  a  sealing 
plug  to  fill  the  stem  at  the  posterior  pole  of  the  globe.  The  plug  includes  an  access  hole, 
filled  with  silicone  adhesive,  which  is  self-sealing  after  puncture.  A  syringe  is  used  to 
evacuate  air  from  the  globe  and  replace  it  with  a  material  representing  the  vitreous 
humor,  a  jelly-like  material  that  fills  the  human  eye  behind  the  lens.  We  have 
employed  a  number  of  different  materials  for  this  purpose  -  polyvinyl  acetate/borax 
“slime”,  egg  white  and  Astroglide  lubricant,  all  of  which  exhibit  a  “stringiness”  when 
an  instrument  contacts  the  material  and  is  withdrawn,  similar  to  the  vitreous  humor. 


Figure  5:  Eye  globe  fabrication  sequence  and  modeling,  example  of  “stringiness”  of  materials  similar  to 
vitreous  humor 
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2.2.  Surgical  Instruments,  Tracking  System  and  Sensors 
2.2.1.  Surgical  Instruments 

In  the  repair  of  lid  (and  later,  globe)  lacerations,  a  variety  of  surgical  instruments  are 
necessary.  For  our  simulator,  we  provide  the  user  with  Castro viejo  suturing  forceps,  a 
Castro viejo  curved  needle  holder  and  curved  Westcott  tenotomy  scissors  (Figure  2 
(right)).  Additional  instruments  are  available  for  future  procedures,  including  finer 
tipped  tools  for  comeo-scleral  injuries.  During  scenarios,  information  on  the 
instruments’  spatial  positions,  open/close  position  and  tip  force,  and  surgeon’s  hand 
motion  is  required  to  grade  the  surgical  techniques  of  users.  To  detect  the  instrument 
motion,  we  use  miniature,  magnetic  position  tracking  sensors  (Model  180  6DOF  sensor, 
Ascension  Technology,  USA)  and  a  tracking  system  (3D  Guidance  trakSTAR, 
Ascension  Technology,  USA).  To  be  compatible  with  the  magnetic  tracking  system,  all 
of  the  instruments  are  fabricated  from  titanium.  The  instruments  are  enhanced  with 
strain  gages  (CEA-06-062UW-350,  Micro-Measurements,  USA),  which  are  placed  at 
appropriate  sites  based  on  finite  element  analysis  (SolidWorks  2012),  to  detect  the 
open/close  position  of  the  instrument  jaws  and  to  measure  the  force  applied  between 
the  jaws  of  the  forceps  to  the  tissue  (Figure  6).  To  convert  the  gage  output  signals  into 
useful  data  for  our  system,  we  developed  miniature  circuit  boards  (actual  size  10mm  x 
16mm)  with  a  microcontroller  (Microchip  PIC18F2450)  and  electrical  components  that 
are  attached  to  each  instrument.  This  is  achieved  using  housings  fabricated  using  the 
3D  printer.  The  microcontroller  supports  USB  communication  and  has  a  10-bit  analog 
to  digital  converter  with  10  different  inputs.  The  board  currently  communicates  with 
the  PC  using  the  USB  bus.  For  each  instrument,  there  are  three  analog  output  signals: 
one  is  the  tool  identification  signal;  the  other  two  are  the  opening  and  force  states  of  the 
tool.  The  tracking  sensors  are  expensive  and  are  tethered  to  their  interface  electronics, 
so  to  minimize  cost  and  the  number  of  cables  that  the  surgeon  must  contend  with,  we 
developed  interchangeable  connectors  between  the  instruments  and  DAQ/interface 
board,  which  also  hold  and  align  the  tracking  sensor  with  the  instruments.  Figure  7 
shows  the  pad  and  component  layout  of  the  DAQ/interface  board.  To  the  left  are  three 
micro-USB  connectors  that  mate  with  the  instrument  mini  boards.  Top  and  bottom  are 
headers  which  allow  connection  to  additional  boards.  To  the  right  are  (top  to  bottom), 
power  supply,  USB  connection,  and  a  chip  programming  interface  connector.  This 
board  can  also  be  used  for  future  motor/pump  control  and  additional  sensor  DAQ. 


Figure  6:  Finite  element  analysis  results  for  half-model  of  each  instrument:  (left)  the  maximum  strain  sites  at 
the  rear  of  the  instruments  are  the  best  place  to  detect  the  open/close  position  of  instruments,  (right)  To 
measure  the  tip  force  we  attached  the  strain  gage  to  the  flat  inner  surface  as  close  as  possible  to  the  tip. 

2.2.2.  AR  hardware 

To  detect  hand  motions  around  and  display  graphical  overlays  onto  the  mannequin 
head,  we  are  using  the  Microsoft  Kinect  motion  tracking  system  and  a  “pico”  projector 
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(Vivitek  Qumi).  We  also  developed  a  portable  framing  system  to  ensure  precise  and 
reliable  registration  between  the  display  and  tracking  systems.  The  projector  and  Kinect 
can  be  adjusted  and  locked  into  the  best  position.  The  head  of  the  mannequin  can 
extend  over  the  edge  of  the  table  for  access  similar  to  a  patient’s  head  extending  past 
the  end  of  an  operating  table.  The  mannequin  is  rigidly  fixed  to  the  stand,  avoiding 
vibration.  A  tray  provides  a  site  for  a  single-handed  instrument  exchange  fixture.  The 
lighting  system  was  implemented  on  the  simulator  stand  to  approximate  the  intensity  of 
surgical  lamps  and  to  minimize  shadows.  All  of  the  parts  can  be  disassembled  for  ease 
of  transportation;  off-site  data  collection  tests  are  planned. 


Figure  7:  DAQ/interface  board  showing  top-side  component  positions  and  board  layout.  To  the  left  are  three 
micro-USB  connectors  that  mate  with  the  instrument  mini-boards.  Top  and  bottom  are  headers  which 
connect  to  additional  boards.  To  the  right  are  (top  to  bottom),  power  supply,  USB  connection,  and  a  chip 
programming  interface  connector  (for  installing  the  firmware  on  the  microcontroller). 

2.2.3.  Modular  eye  motion  and  pupil  contraction  systems 

Future  trauma  scenarios  will  require  additional  functionality  in  the  replaceable 
lid/globe  modules,  so  we  are  developing  eye  motion  and  pupil  contraction  components. 
These  motions  are  intended  for  first-responder  scenarios,  in  which  they  can  provide 
clues  during  diagnosis  of  the  casualty  in  an  emergency  incident.  As  space  is  limited 
within  a  simulator’s  head,  and  noise  from  some  types  of  motors  may  be  distracting  to 
the  user,  we  are  considering  the  use  of  shape  memory  alloy  (SMA)  wire  actuation. 

In  one  prototype,  the  eye  motion  component  consists  of  two  SMA  linear  actuators 
(MigaOne-15,  MIGA  Motor  Company),  a  driver  and  motion  limit  detection  circuit,  a 
pivoting  eye  motion  plate,  ball-and-socket  links,  connectors,  and  supporters  as  shown 
in  Figure  8  (left).  The  circular  plate  rotates  in  two  degrees  of  freedom  (pitch  and  yaw). 


Figure  8:  (left)  2DOF  rotation  eye  motion  and  (right)  pupil  contraction  components  using  SMA  actuators. 

In  order  to  develop  the  pupil  contraction  component,  we  observed  the  range  of 
motion  of  pupil  contraction  and  dilation,  and  designed  a  contracting  iris  mechanism. 
Figure  8  (right)  shows  a  silicone  iris,  supporting  parts,  fibers  linking  the  iris  to  the 
actuator,  and  an  SMA  linear  actuator  (NanoMuscle  NM706-Super,  MIGA  Motor 
Company).  We  sutured  the  inner  margin  of  the  silicone  iris  using  thread,  which  passes 
through  an  array  of  holes.  When  the  SMA  actuator  contracts,  the  fibers  pull  the  inner 
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iris  margin  towards  the  center  of  the  pupil.  When  the  actuator  is  shut  off,  the  elastic  iris 
returns  to  its  normally  dilated  position. 

2.3.  Content  and  learning  architecture 

Our  initial  developments  towards  a  content  presentation  and  learning  architecture  is 
geared  towards  junior  surgeons  with  limited  prior  oculoplastics  and  globe  surgery 
experience,  and  more  senior  physicians  who  may  not  have  frequent  trauma  patient 
exposure.  We  aim  to  tailor  the  system  to  military  surgeons,  who  would  benefit  from 
pre-deployment  ocular  and  periocular  surgical  planning  and  technique  refresher  courses. 

The  planned  architecture  will  allow  access  to  background  content  and  teaching 
materials  on  multiple  platforms.  Smart-phones,  personal  computers  and  similar  devices 
would  be  appropriate  at  this  stage.  The  user’s  interaction  will  begin  with  a  typical  login, 
in  which  the  system  recognizes  the  identity  and  level  of  expertise  of  the  user.  This  will 
be  followed  by  content  access,  logged  by  the  system  to  allow  bypass  of  such 
background  when  access  to  the  simulator  is  available.  In  the  current  context,  this  would 
include  relevant  anatomy,  suturing  technique,  operative  strategy  and  surgical  tips. 

As  mentioned,  our  first  scenario  is  marginal  lid  laceration,  although  the  current 
module  design  also  supports  non-marginal  injury  presentation.  Injuries  are  generated 
by  manually  cutting  through  the  skin  and  tarsal  plate  structures  of  the  module  in  the 
location  desired  by  the  trainer/course  proctor.  At  the  outset  of  a  simulator  session, 
patient  history,  lab  tests,  pre-op  imaging  work  and  similar  information  will  be 
presented  to  the  user.  Once  these  items  have  been  reviewed,  the  simulator  is  initialized, 
the  user  takes  hold  of  selected  instruments,  and  the  trauma  module  is  unveiled. 

The  user,  with  instrument  and  hand-tracking  enabled,  will  perform  an  examination 
of  the  injury,  verify  the  absence  of  foreign  bodies  and  additional  injuries  (in  the  initial 
scenario),  and  begin  to  conduct  the  repair.  Tracking  and  sensor  data  are  recorded  to 
disk  for  post-processing  at  present,  however  we  are  developing  online  gesture 
recognition  techniques  to  allow  intra-operative  feedback,  such  as  suggestions  when  a 
non-standard  technique  appears  to  be  used. 

During  the  scenario,  the  user  will  be  able  to  use  two  or  three  instruments 
simultaneously  (surgeons  often  palm  one  instrument  and  pick  up  an  additional  one  in 
the  same  hand  before  returning  it  to  the  tray  -  cutting  of  excess  suture  is  one  typical 
example,  in  which  forceps  or  a  needle  holder  are  not  put  down  first),  and  the  tracking 
system  will  be  able  to  identify  which  instrument  is  in  use  at  a  given  time.  For 
procedures  requiring  four  or  more  instruments,  the  user  will  hot-swap  the  connector  on 
the  end  of  an  instrument  no  longer  in  use,  to  the  newly  necessary  device.  The 
connector/D AQ  system  automatically  detects  its  identity. 

Following  scenario  completion,  performance  statistics  will  be  presented  to  the  user. 
At  present,  we  can  calculate  task  completion  time  and  path  length,  as  well  as  timing 
and  sequence  of  instrument  usage.  We  are  developing  techniques  to  identify  distinct 
surgical  gestures  (e.g.  knot  tying,  suture  cutting,  tissue  grasping),  so  that  finer  grained, 
more  useful  feedback  can  be  determined  and  presented  to  the  user. 

2.4.  Data  Acquisition  and  Event-based  Gesture  Segmentation  Software 

At  present,  software  developments  have  primarily  focused  on  data  acquisition  and 
instrument  interface  tools.  The  application  has  the  flexibility  to  use  multiple  user 
interface  input  devices  (e.g.  mouse,  SpaceNavigator,  trackball).  It  records  position 
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tracking  streams  from  up  to  eight  different  magnetic  position  tracking  sensors,  from  up 
to  two  strain  gages  on  each  instrument,  and  the  identity  of  each  instrument.  In  addition, 
Kinect  outputs,  representing  hand  motions  (vs.  instrument  motions)  are  being 
integrated  into  the  system;  hand  tracking  can  be  used  to  detect  actions  like  picking  up  a 
specific  type  of  suture.  All  of  these  data  are  recorded  as  separate  streams  and  saved  in 
XML  format.  Simultaneously,  a  3D  rendering  of  the  instruments  and  head  form  is 
displayed  to  the  user  (Figure  9).  The  system  is  easily  expandable  to  track  more 
instruments,  the  AR  loupes  when  available,  and  head  motion  relative  to  the  base  once  a 
flexible  neck  is  implemented.  The  3D  rendering  is  important  as  it  enables  navigation 
around  the  scene,  useful  during  the  post  processing  playback  as  we  manually  observe 
task  performance  and  work  towards  an  automated  gesture  recognition  algorithm. 

Thresholding  functions  are  built  into  the  DAQ  software,  allowing  preliminary 
segmentation  using  features  such  as  closing  and  opening  of  scissors  (indicating  cutting), 
and  the  simultaneous  closure  of  forceps  jaws  and  the  increase  of  measured  force, 
suggesting  that  tissue  has  been  grasped  (vs.  closing  in  free-space).  Similarly,  the  entry 
into  or  exit  from  pre-determined  regions  of  interest  (Rol’s)  are  recognized  as 
meaningful  state  transitions.  We  refer  to  this  as  “event-based  surgical  gesture 
recognition,”  and  propose  that  the  subsequent  recognition  of  sequences  of  events  will 
be  useful  for  identifying  differences  in  performance  between  users.  This  process  will 
be  described  in  detail  in  a  separate  paper. 


Figure  9:  (left)  Developer's  interface  showing  data  streams  and  graphical  display  of  instrument  positions, 
(right)  Data  stream  examples  showing  discrete  events  to  be  segmented  using  software  under  development. 


3.  Data  Acquisition  Trials 

Three  of  our  subject  matter  expert  oculoplastics  surgeons  have  performed  marginal 
laceration  repair  on  our  modules,  in  each  case  repairing  cuts  in  nearly  identical 
locations  and  of  identical  geometry  (perpendicular,  full-thickness  upper  lid  laceration  at 
lid  mid-line,  with  extension  of  the  laceration  parallel  to  lid  margin).  They  were  asked 
to  perform  the  repair,  aligning  and  connecting  the  tarsal  plate  and  overlying  tissues, 
while  making  only  commentary  that  would  typically  be  employed  in  the  OR  (rather 
than  in  a  teaching  case,  in  which  additional  discussion  would  be  necessary).  Following 
the  repair  exercise,  they  performed  a  series  of  15  partial  tasks,  including  a  variety  of 
knot  tying  and  suturing  techniques,  to  help  us  collect  data  in  support  of  detecting  the 
discrete  events  within  the  extended  full  scenario  testing. 

In  all  of  these  tasks,  instrument  tracking  was  recorded,  as  was  video-taping  of  the 
operative  field,  and  an  external  observer’s  synchronized,  manual  event  detection  (time- 
stamped  key  presses).  Analysis  of  these  data  is  underway;  the  analysis  techniques  are 
described  in  an  accompanying  paper  [16].  The  results  will  be  used  to  automate  gesture 


M.P  Ottensmeyer  et  al.  /  Ocular  and  Craniofacial  Trauma  Treatment  Training  System 


329 


recognition  during  external  user  data  collection  trials  in  the  next  few  months  (these 
studies  will  be  covered  under  appropriate  human  subjects  research  approvals  from  our 
IRB  and  that  of  the  funding  sponsor).  Those  tests  will  include  both  junior  and  senior 
surgeons,  and  we  expect  to  find  distinguishing  characteristics  between  them  that  will 
allow  us  to  provide  meaningful  and  timely  scoring  and  recommendation  to  future  users. 


4.  Results 

At  present,  we  have  a  functional  system  that  is  in  use  by  our  subject  matter  experts  for 
initial  testing  purposes.  It  is  capable  of: 

•  Real-time  on-screen  display  of  surgical  motions  with  animated  instruments 

•  Display  of  all  trajectory,  force,  grasp  position  values  in  real  time 

•  Detection  of  thresholds  to  identify  grasp/release 

We  have  identified,  through  the  experts’  usage  experience,  a  series  of 
improvements  to  the  eye  lid  structure  (addition  of  tarsal  plate  to  original,  single  layer 
lid,  corrections  to  the  material  properties  of  the  tarsal  plate  and  skin  silicone  selection). 
Similarly,  the  instrument  connectors  and  cable  management  have  improved,  with 
additional  sensor  attachment  stability,  strengthened  mating  between  the  connector 
housing  and  the  instruments,  and  modifications  to  the  Velcro  attachments  we  use  to 
attach  sensor  wires  to  the  user’s  wrists  (and  out  of  the  way  of  the  instruments). 

Most  importantly,  initial  data  collections  are  allowing  us  to  proceed  towards  the 
development  of  gesture  segmentation  algorithms  and  scoring  methods. 

Informal  feedback  from  our  surgeon  experts  suggest  that  we  are  close  to 
acceptably  realistic  tissue  stiffness,  both  around  the  lid  region  and  around  the  rest  of  the 
head  form.  Similarly,  they  approve  of  the  globe  geometry,  layer  thicknesses,  cornea 
and  lens  clarity,  and  properties  of  the  vitreous  humor. 


5.  Future  work 

We  are  progressing  towards  a  complete  system  on  a  number  of  different  fronts. 

The  eyelid  anatomy  continues  to  evolve  as  we  implement  additional  realism. 
Meibomian  gland  pores,  adjustments  to  layer  thickness,  and  the  addition  of  the  lacrimal 
ducts  and  canthal  ligaments  are  under  development,  as  are  methods  for  including 
conjunctival  and  muscle  layers. 

Revisions  to  the  signal  acquisition  and  conditioning  circuitry  are  under  way  -  the 
current  “mini-boards”  exhibit  some  heating  in  the  circuit  that  is  noticeable  to  the  user 
and  may  be  affecting  the  gain  and/or  offset  of  the  force  and  grasp  position  signals. 

The  pupil  contraction  and  ocular  motion  components  are  being  integrated  into  a 
modified  version  of  the  existing  globe  anatomy,  and  additional  sensors  to  detect  the 
patency  of  repairs  to  comeo-scleral  lacerations  will  be  added  to  the  system.  These 
sensors  will  also  be  useful  to  aid  in  the  detection  of  canthotomy/cantholysis  treatment 
in  response  to  simulated  retrobulbar  hemorrhage. 

Beyond  improvements  to  the  current  system,  the  head  form  will  be  reconfigured  to 
accept  multiple  eye  portals  (left  and  right,  and  different  trauma  states),  as  well  as  facial 
hemorrhage  and  fracture  modules,  and  airway  components,  which  will  support  the 
detection  of  artificial  airway  devices  and  surgical  cricothyroidotomy. 
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Abstract.  This  paper  describes  a  novel  approach  for  segmenting  surgical  gestures 
in  an  eye  trauma  treatment  simulator,  based  on  the  analysis  of  system  and 
environmental  events,  generated  in  order  to  track  user  performance.  Subdivision  of 
surgical  procedures  into  tasks  &  sub-tasks  allows  for  application  of  more  accurate 
metrics  (optimized  per  task),  which  may  more  appropriately  evaluate  user 
performance.  This  approach  may  also  be  applied  in  other  surgical  contexts.  For  the 
specific  case  of  eye  trauma  treatment  this  system  can  generate  different  classes  of 
events  that,  when  combined,  may  drive  a  state  machine  with  the  capability  of 
subdividing  the  macro  operation(s).  The  results  of  this  process  may  be  utilized  to 
develop  informative  feedback  to  the  user  in  the  form  of  performance  metrics  and 
training  guidance. 


Keywords.  Surgical  gesture  recognition,  surgical  simulation,  surgical  training, 
evaluation  metrics. 


Introduction 

One  of  the  most  important  aspects  of  surgical  training  tools  is  providing  accurate  and 
appropriate  evaluation  of  user  performance.  This  evaluation  is  a  non-trivial  process 
with  many  facets  and  approaches  [1][2][3][4].  In  the  context  of  surgical  gesture2, 
performance  is  commonly  evaluated  through  the  analysis  of  movements  and  trajectories 
obtained  from  tracking  the  path  of  the  surgical  instruments,  and  their  associated 
geometrical  (spatial),  and  temporal  features  (e.g.  smoothness,  speed,  and  compactness) 
[4].  One  method  to  improve  this  evaluation  process  is  to  reduce  the  overall  surgical 
procedure  into  a  set  of  tasks  and  to  apply  specific,  task-oriented,  evaluation  metrics  to 
each  task  [1].  Another  important  rationale  for  conducting  this  reductive  process  is  that 
it  permits  selective  exclusion  of  unrelated  or  extraneous  actions  (e.g.  environment 
interactions)  from  potentially  computation-intensive  analyses.  One  of  the  issues  related 
with  the  task  evaluation  approach  is  that  the  entire  procedure  must  be  subdivided  into 

1  Corresponding  author. 

2 

A  surgical  gesture  is  the  execution  of  a  specific  action,  performed  by  moving  and/or  acting  on  the 
surgical  tools. 
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tasks,  subtasks  and  gestures  automatically  -  computationally  analyzing  the  movement 
data  to  identify  each  gesture,  subtask,  or  task  (gesture-subtask-task  segmentation)  [6]  [9]. 
A  significant  body  of  work  has  been  devoted  to  this  last  topic.  For  example,  research 
has  been  conducted  based  on  evaluating  the  human  body  movements  to  analyze 
surgical  gestures  using  human  anatomy-driven  hidden  Markov  models  (HMM)  [7]. 
While  the  most  common  approaches  are  movement-based,  gesture  segmentation  does 
not  necessarily  need  to  be  driven  by  analysis  of  the  surgeon’s  (or  surgical  instruments) 
movements.  The  movement  signal  is  necessary  in  order  to  identify  the  level  of 
expertise  of  the  surgeon.  Efficient  movements  made  by  an  experienced  surgeon  are 
typically  more  accurate  and  localized  within  the  surgical  field  than  dispersive  and  less 
compact  movements  made  by  novices  [9]. 

This  paper  describes  a  new  approach  for  surgical  gesture  segmentation  and  task 
recognition  that  can  be  performed  by  analyzing  the  events  generated  by  the  system 
(surgical  instruments  and  environment)  instead  of  the  surgeon’s  (or  surgical 
instruments)  movements.  The  key  motivation  for  focusing  attention  on  the  events 
generated  by  the  system  is  that  the  events  may  contribute  significant  intermediate 
representations  of  information.  This  serves  to  discretize  the  movement  recognition  and 
analysis  process,  thereby  reducing  the  complexity  of  segmentation.  We  define  events  as 
actions  that  can  change  the  state  of  a  state  machine,  e.g.  opening  and  closing  of  a 
specific  instrument,  grasping  with  forceps,  or  entering/exiting  from  a  specific  region- 
of-interest.  Assuming  a  well-structured  system,  it  is  possible  to  track  the  appropriate 
number  and  class  of  events.  From  this  point,  it  may  be  possible  to  identify  specific 
patterns  of  events/states  that  clearly  correlate  with  specific  gestures,  subtasks  and  tasks 
of  a  surgical  procedure. 

The  focus  of  this  paper  is  the  application  of  this  event-driven  approach  in  an  ocular 
trauma  simulator.  However,  this  event-driven  approach  may  be  extended  to  a  wider 
range  of  surgical  procedures. 


1.  Background 

In  2003,  our  group  developed  a  laparoscopic  skills  trainer  based  on  the  SAGES 
Fundamentals  of  Laparoscopic  Surgery  (FLS)  task  curriculum.  Using  a  modified 
commercial  tracking  system,  the  Computer-Enhanced  Laparoscopic  Skills  Training 
Simulator  (CELTS)  allowed  trainees  to  use  actual  laparoscopic  instruments  to  perform 
FLS  tasks  on  physical  “task  boards”  including  a  silicone  suture  pad  and  peg-boards. 
The  CELTS  system  resulted  in  the  formation  of  several  fundamental  design  concepts 
that  we  have  leveraged  in  our  current  ocular  and  craniofacial  trauma  simulator.  CELTS 
implements  a  full-color,  real-time  video  display,  rather  than  virtual  rendering,  to 
provide  the  surgical  field  of  view.  In  combination  with  natural  gravitation  effects  and 
standard  laparoscopic  instruments,  CELTS  provided  trainees  with  a  realistic 
approximation  of  procedural  elements.  The  specialized  tracking  system  permitted 
recording  of  translation  and  rotation  with  additional  software -based  task  metrics.  Using 
this  system,  our  group  demonstrated  that  appropriate  performance  metrics  could  be 
defined  and  a  standardized  scoring  system  could  be  designed  [9]. 

The  metrics  generated  by  CELTS  involved  post-processing  of  the  entire  motion 
trajectory  of  the  surgeon’s  gestures,  which  illustrated  distinct  differences  in 
performance  between  experts  and  novices.  However,  fmer-grained  details  used  to 
provide  real-time  feedback  and  automated  generation  of  suggestions  for  improvement, 
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could  not  be  extracted  using  those  metrics  alone.  To  address  this  issue,  we  propose  a 
method  of  event-based  task  segmentation  that  will  enable  immediate  evaluation  and 
improved  feedback. 


2.  Methods 


The  methodology  utilized  for  surgical  gesture  segmentation  and  task  recognition  is 
driven  by  specific  patterns  of  events  generated  by  the  system.  By  this  process,  motion 
path  analysis3  is  deferred  to  a  conditional,  later  tier  of  analysis.  The  analysis  of  a  list  of 
discrete  events  is  considerably  less  computationally  expensive,  than  the  continuous 
analysis  on  a  moving  window  of  data  sampled  at  high  rates  from  one  or  more  surgical 
instruments  paths.  This  may  unambiguously  indicate  user  actions  and  in  cases  where  it 
does  not,  additional  path  analysis  can  be  conducted  on  a  focused  subset  of  the  streamed 
data.  The  same  approach  may  be  applied  both  to  the  subtask  and  task  levels  of  analysis. 
As  in  other  methodologies,  the  first  step  to  perform  is  to  determine  the  structure  of  a 
data  frame  of  measured  parameters  at  each  time  point.  In  our  case,  the  parameters 
measured  for  each  surgical  instrument  were:  time  stamp;  instrument  identity;  raw 
sensor  position  and  orientation;  jaw  closure;  and  jaw  grasping  force.  Second  order  data 
extends  the  data  frame  by  deriving  instrument  tip  position,  tip  velocity,  and  tip 
acceleration. 


Based  on  the  data  types  available,  we  define  sets  of  thresholds.  Events  are  defined 
as  threshold  crossings.  Observing  the  data  frames  over  time,  the  system  creates  a  list  of 
time-stamped  events,  such  as  the  moment  when  a  spatial  region  of  interest  (ROI)  is 
entered  or  an  instrument’s  jaws  are  fully  closing.  This  list  of  events  would  typically  be 
orders  of  magnitude  smaller  than  the  original  data  set. 

From  a  detailed  task  analysis  of  relevant  surgical  scenarios,  we  define  gestures  as 
meaningful  patterns  of  events.  The  system  creates  recursively  higher  order  lists  of 
gestures,  subtasks,  tasks,  and  ultimately,  procedures,  corresponding  with  the  different 
levels  of  the  tasks  analysis  hierarchy  (Table  1, Figure  1). 

Identifying  actions  at  each  level  of  the  taxonomy  provides  the  opportunity  to 
provide  the  best  evaluation  metrics  at  the  most  appropriate  moment  during  the  scenario. 
Depending  on  the  training  goals,  this  may  be  evaluation  of  decision  processes, 
confirmation  of  correct  sequencing  of  sub-tasks  or  gestures,  or,  when  required,  detailed 
analysis  of  a  subset  of  the  motion  data  within  a  gesture.  At  the  same  time,  it  allows 
exclusion  from  the  evaluation  process  of  extraneous  motions  (e.g.  interacting  with  other 
colleagues  or  with  the  environment). 


Table  1.  System  Taxonomy 


Classification 

Definition 

Examples 

First  order  data 

Raw  sensor  data 

•  Instruments  positions/orientations 

•  Instmments  opening/closing 

•  Instruments  force 

Second  order 

Data  derived  from  raw  sensor 

•  Instruments  tip  positions 

data 

data 

•  Instrument  velocity 

Recognition  of  specific  paths,  curves  and  other  features  by  specific  features  extracted  from  the  path 
followed  by  the  surgical  tool. 
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Event 

A  crossing  of  a  defined 
threshold  for  a  given  stream  of 
sensor  data  or  second  order  data 

•  Tip  of  needle  holder  enters  a  ROI. 

•  Forceps  open 

•  Scissor  cut 

Gesture 

A  pattern  of  events 

•  Needle  holder  tip  is  closed. 

•  Forceps  grasp  with  medium  tension 

•  Needle  holder  pitches  95  degrees  about  its  axis. 

•  Cut  w/  scissors 

Subtask 

A  pattern  of  gestures 

•  Pass  Needle 

•  Two-throw  knot  tie 

•  One -throw  knot  tie 

Task 

A  set  of  subtasks 

•  2-1-1  knot  placement 

•  Interrupted  suture 

•  Running  suture 

•  Exploration  w/  forceps 

Procedure 

A  set  of  tasks 

•  Full  thickness  eyelid  laceration  repair 

Events  FIFO  Gestures  List  Subtask/tasklist 

Figure  1.  Organization  structure  of  taxonomy. 


3.  Experimental  Setup 

The  current  approach  was  evaluated  using  the  ocular  trauma  surgical  simulator  system 
that  we  are  developing,  described  briefly  below  (Figure  2).  The  system  is  described  in 
more  detail  in  an  accompanying  paper  [10]. 
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Figure  2.  Simulation  platform:  mannequin  head,  instrument  tray,  tracking  system,  lighting  system,  overhead 
projector  and  Kinect  tracker. 


The  platform  is  based  on  a  silicone  head  attached  to  a  vertical  stand  supporting  a  3D 
camera  (Microsoft  Kinect™)  used  to  track  the  interaction  between  the  user  and  the 
simulation  content,  a  set  of  lights,  and  a  video  projector.  The  mannequin  head  includes 
a  set  of  interchangeable  portals  designed  to  simulate  a  range  of  trauma  procedures 
(Figure  3,  left).  We  are  currently  focusing  on  events  related  to  repair  of  eyelid 
lacerations. 


Figure  3.  (left)  Eye  trauma  portals  and  top  view  of  instruments  with  position  tracking  sensors  attached  (right). 


The  system  provides  the  user  with  a  set  of  instruments  necessary  to  perform  relevant 
surgical  tasks.  Each  instrument  is  connected  to  a  magnetic  tracking  system  (Figure  3, 
right),  which  measures  instrument  positions  and  orientations  in  six  degrees  of  freedom 
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(DOF),  and  to  a  sensor  circuit  able  to  measure  the  opening/closing  dynamics  at  the 
instrument  tip  and  the  force  applied  to  the  tip  of  the  instrument.  The  sensors  used  on  the 
surgical  instruments  are  a  set  of  strain  gauges  connected  with  a  signal  conditioning 
circuit  board. 

A  series  of  opthalmic  surgical  instruments  were  connected  to  the  magnetic  tracking 
system  for  this  evaluation  (Figure  4) 


Figure  4.  Surgical  instruments  set:  1-510  Westcott  scissors  curved  tip,  2255T  Castroviejo  forceps,  4536T 
Castroviejo  needle  holder. 

The  system  is  controlled  by  data  acquisition  software  (Figure  5,  left),  which  provides 
capabilities  to  record  data  provided  from  the  instruments/sensors  and  other 
environmental  parameters.  The  Graphical  User  Interface  (GUI)  also  provides  real-time 
3 -dimensional  (3D)  displays  of  the  mannequin  and  instrument  positions,  as  well  as 
configurable  multi-track  views  of  the  recorded  data. 


Figure  5.  (left)  GUI  of  data  acquisition  and  analysis  software,  (right)  Regions  of  Interest  (ROIs)  and 
instrument  interactions  with  ROIs. 

A  significant  feature  provided  by  the  software  tool  is  the  management  of  multiple 
spherical  regions-of-interest  (ROI),  allowing  the  creation  of  maps  and  generating 
events  for  each  crossing  of  an  ROI  boundary  (Figure  5,  right). 
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The  software  records  the  signals  coming  from  the  system  (instrument 
position/orientation/state,)  and  displays  this  information  in  a  timeline.  To  support  the 
experimental  design  below,  the  interface  also  provides  the  capability  to  insert  time- 
stamped  markers,  using  an  external  interface  device  (mouse,  keyboard,  etc).  The  listing 
of  these  keystrokes  is  shown  in  Table  2. 

Table  2.  Direct  Observation  Markers 

0.  Null  key 

1 .  Needle  point  pierces  tissue 

2.  Needle  eye  retrieved  from  tissue 

3.  Knot  throws  initiated 

4.  Knot  pulled  taught 

5.  Scissors  retrieved  from  /  returned  to  tray 

6.  Cut  with  scissors 

7.  Non-tracked  hand  action 

8.  User  error  or  digression 

9.  Verbal  comment 


4.  Experimental  Design 

The  current  prototype  allowed  collection  of  natural  human  interaction  data  in  a  highly- 
structured  fashion  to  support  initial  investigation  into  our  theory  of  event-driven  gesture 
segmentation  and  subsequent  task  recognition. 

To  develop  an  understanding  of  how  to  parse  the  data  collected  by  the  simulator,  it 
was  initially  necessary  to  identify  what  parameters  constitute  an  event.  Identification  of 
the  unique  types  of  these  events  throughout  a  surgical  procedure,  currently  focusing  on 
suturing  of  an  eyelid  laceration,  was  also  required. 

4.1.  Experimental  apparatus 

The  subject  matter  experts  (SMEs)  were  provided  with  one  or  two  pairs  of  surgical 
gloves,  depending  on  personal  preference.  Several  varieties  of  suture  and  needle 
combinations  were  available  to  provide  the  SMEs  with  familiar  interventional  options. 
Across  our  three  subject  matter  experts,  preferences  were  for  Ethicon  5-0  Perma-Hand 
silk  with  P-3  needles,  Ethicon  6-0  Vicryl  with  P-3  needles,  and  Ethicon  6-0  Prolene 
sutures  with  P-3  needles.  Task  portals  were  inserted  into  the  mannequin  head,  synthetic 
blood  was  applied  to  the  wound,  and  the  lacerated  eye  was  covered  with  a  Fox  eye 
shield  to  hide  the  simulated  injury  until  the  beginning  of  the  scenario.  The  prototype’s 
LED  lighting  units  were  turned  on,  and  were  augmented  with  a  Storz  fiber  optic  Xenon 
surgical  lamp,  per  the  SME’s  request. 

4.2.  Protocol  for  Capturing  Expert  Data  During  a  Simulated  Eyelid  Laceration  Repair 

In  order  to  keep  the  SME’s  performance  as  natural  as  possible,  the  protocol  was 
designed  to  minimize  simulator-specific  motions.  Precautions  were  taken  to  avoid 
having  the  SME  modify  his  or  her  natural  gestures  to  accommodate  discrepancies  in  the 
simulated  procedure.  Brief  conversations  with  the  engineers,  comments  on  known 
issues  with  the  prototype,  and  a  general  hesitation  to  break  potentially  fragile 
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components  could  have  distorted  how  an  SME  would  use  the  simulator.  Surgeons  are 
often  also  natural  teachers  and  tend  to  explain  their  actions  during  the  data  capture 
process.  Therefore,  sessions  began  with  strict  guidance  not  to  offer  comments,  illustrate 
thoughts  by  pointing  with  instruments,  or  teach  using  the  simulator.  The  SMEs  were 
instructed  to  remove  the  Fox  eye  shield,  investigate  the  wound,  and  treat  the  injury  as 
they  would  in  real  life.  To  ensure  the  session  was  as  unbiased  as  possible,  SMEs  were 
asked  to  complete  the  full  eyelid  repair  before  focusing  on  repetitive  tasks  so  as  to 
avoid  any  simulator-specific  learned  behaviors.  In  addition  to  the  unstructured  surgical 
performance,  the  structured  exercises  recorded  across  all  three  experts  were  as  follows: 

1 .  Grasp  suture  from  tray  and  load  the  needle  in  the  needle  holder. 

2.  Double-bite  through  both  sides  of  incision,  pull  length  of  suture  through 

3.  Double-bite  with  2-1-1  surgeon’s  knot,  excess  cut  with  scissors 

4.  Single-bite  through  both  sides  of  incision,  pull  length  of  suture  through 

5.  Single-bite  with  2-1-1  surgeon’s  knot,  excess  cut  with  scissors. 

6.  Suture  placed,  pull  length  of  suture  through  until  ready  to  tie  knot 

7.  Suture  placed,  tying  of  1-1-1  surgeon’s  knot. 

8.  Suture  placed,  tying  of  2-1-1  surgeon’s  knot. 

9 .  Suture  placed,  tying  of  3  - 1  - 1  surgeon  ’  s  knot. 

10.  Suture  placed,  tying  of  3-1  granny- 1  adjustable  surgeon’s  knot. 

1 1 .  Grasp  tissue  with  forceps,  light  pressure. 

12.  Grasp  tissue  with  forceps,  normal  pressure. 

13.  Grasp  tissue  with  forceps,  heavy  pressure. 

14.  Vertical  mattress  suture. 

15.  Running  suture. 

Three  eye  portals  were  fabricated  from  our  multi-layer  multi-part  molds  that  shared  a 
similar  laceration  on  the  upper  eyelid.  The  laceration  was  a  full-thickness  wound  and 
was  generally  repaired  with  6  to  8  interrupted  sutures.  Figure  6  shows  the  repairs  and 
the  sutures  that  were  placed  during  the  later  discrete  task  sessions.  Minimally  guided 
performance  by  all  three  SMEs  was  found  to  be  similar. 

The  system  was  configured  to  collect  data  from  the  SMEs  on  both  a  complete 
procedure,  as  well  as  a  specific  set  of  tasks.  As  the  SMEs  were  performing  both  the 
open-ended  eyelid  laceration  repair  and  the  discrete  event  tasks,  an  observer  manually 
recorded  the  data  captured  by  using  the  keyboard  attached  to  the  simulator.  These 
manual  keystrokes  facilitated  the  later  synchronization  of  the  multiple  sources  of  data: 
overhead  video,  3D  path  data,  and  the  initial  event  recognition  output.  The  manual  keys 
were  used  when  parsing  the  path  data  to  begin  to  identify  trends,  distinct  markings,  and 
patterns  while  helping  to  ignore  erroneous  data. 
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Figure  6.  Silicone  eye  portals  illustrating  the  final  suture  patterns  created  by  three  different  ocular  surgeons. 


5.  Preliminary  Data,  Analysis  and  Discussion 

The  analysis  below  focuses  on  Exercise  #8:  the  tying  of  a  2-1-1  surgeon’s  knot.  The 
raw  data  collected  by  the  simulator  was  parsed  to  begin  to  extract  events  that 
characterized  this  known  task.  Each  track  of  data  was  analyzed  using  three  different 
methods:  visualization  of  the  raw  data,  characterization  of  the  recorded  video,  and 
normalization  across  all  three  sessions  for  each  of  the  three  SMEs. 

Figure  7  shows  the  open/close  position  data  collected  from  each  of  the  three  SMEs, 
overlaid  with  the  corresponding  manually  identified  needle  grasp  and  release  gestures. 
One  can  identify  correspondence  between  the  sensor  data  and  the  observations,  which 
supports  the  proposition  that  an  automated  event  detection  and  gesture  recognition 
system  can  reproduce  the  manual  observations. 
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Figure  7.  Raw  needle  holder  open/close  position  data  and  overlaid  observational  event  markings. 

Figure  8  shows  a  schematic  representation  of  the  spatial  ROIs  around  the  eye.  The 
radii  of  these  spheres  and  their  centers  were  established  in  relation  to  the  real  anatomy, 
and  the  raw  position  data  processed  to  identify  the  boundary  crossing  events  for  each 
instrument.  The  time  evolution  of  which  ROI  a  given  instrument  occupies  is  shown  in 
Figure  9,  which  presents  the  motion  of  scissors,  from  occupying  the  instrument  tray  to 
cutting  a  suture  and  returning.  The  sequence  of  events  can  be  interpreted  as  the 
gestures:  (A)  “removal  from  tray,”  (B)  “entry  into  working  space,”  (C,C)  “cutting  of 
suture  ends,”  (D)  “exit  from  working  area”  and  (E)  “return  to  tray.”  Over  multiple 
SMEs  and  multiple  repetitions,  this  sequence  of  gestures  was  the  same. 
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Figure  8.  Regions  of  Interest  (ROI)  defining  areas  where  the  simulator  begins  to  monitor  instrument 
interaction  to  extract  events  for  gesture  segmentation: 


o 

o 

in 


Figure  9.  Use  of  scissors  to  cut  suture  ends 

Using  different  combinations  of  event  sequences  and  their  derived  gesture  sequences, 
higher  levels  of  the  taxonomy  can  be  identified,  such  as  creating  a  2-1-1  surgeon’s 
knot.  The  full  sequence  is  shown  in  Figure  10. 

Both  the  forceps  and  the  needle  holder  start  within  a  relatively  small  ROI  near  the 
eye.  During  the  process,  the  needle  holder  approaches  more  closely,  entering  a  smaller 
ROI.  Presence  within  the  smallest  ROI  and  the  closure  of  the  needle  holder  identifies 
the  grasping  gesture.  The  subsequent  opening  and  closings  are  releases  and  re-grasping 
gestures  as  the  needle  passes  through  the  tissue.  The  departure  of  the  instruments  from 
the  nested  ROIs  suggests  pulling  the  needle  to  draw  the  suture  through  the  skin,  and  in 
combination  with  orientation  thresholds  (not  shown)  the  forceps  are  palmed,  pointing 
away  from  the  suture  site,  hence  apparently  moving  away,  through  the  sequence  of 
ROIs. 

The  re-approach  and  movement  into  and  out  of  the  small  ROIs,  followed  by  a 
needle  holder  closure  (vs.  grasp)  signal  the  double  wrap  of  suture  around  the  needle 
holder  and  closing  of  the  holder  about  the  short  end  of  the  suture  to  form  the  knot, 
which  is  tightened  at  10  seconds  with  the  next  passage  outwards  through  the  ROIs.  The 
close-up  closure  and  release  gestures  signal  single  wraps  of  suture  and  short  tightening 
of  the  next  two  knot  ties.  Finally,  the  needle  holder  passes  out  of  the  eye  centered  ROIs 
and  is  palmed  (orientation  events  not  shown)  and  enters  the  instrument  tray  ROI,  where 
the  scissors  are  picked  up  brought  inwards  to  cut  the  long  end  of  the  suture,  to  be 
returned  and  exchanged  with  the  needle  holder  again.  Together,  these  gestures 
constitute  the  2-1-1  knot  placement  subtask. 
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Figure  10.  Full  sequence  of  needle  insertion  and  tying  of  2-1-1  knot  subtask. 

This  example  presents  the  sequential  re-composition  of  the  raw  data,  events  detected 
and  gestures  into  a  subtask.  Longer  sequences  would  be  further  composed  into  tasks 
making  up  a  procedure.  Similar  observations  of  sets  of  expert  performances  will  be 
parsed  into  the  broad  taxonomic  library  that  the  system  under  development  will  use  to 
segment  automatically  trainee  performance  data  sets,  to  identify  where  they  perform 
sequences  similarly  to  the  experts,  where  and  how  they  differ,  and  ultimately  provide 
feedback  in  support  of  the  learning  process. 
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6.  Conclusion  /  Ongoing  Work 

This  paper  presents  the  early  stage  development  of  a  novel  approach  towards  efficient 
and  useful  surgical  task  recognition  and  performance  measurement.  It  reduces 
computational  load  compared  with  real-time  motion  trajectory  analysis  by  identifying 
and  operating  on  small  numbers  of  discrete,  low-level  events,  which  are  rebuilt  into 
progressively  higher  level,  meaningful  taxonomic  structures.  This  tiered  system  will 
permit  the  identification  of  key  elements  for  evaluation,  and  trigger  feedback 
appropriate  to  the  level  of  analysis. 

The  next  phases  of  the  work  focus  on  characterizing  the  remainder  of  subtasks  that 
the  SMEs  performed;  extracting  additional  thresholds  to  build  the  event  library  and  the 
subsequent  gesture  and  sub-task  sequence  libraries;  validation  of  the  analysis 
algorithms  by  comparing  automated  segmentation  against  manual  recognition;  and 
collecting  extensive  data  sets  from  experts  and  novices  to  begin  to  distinguish  how 
these  two  groups  differ  and  when  those  differences  rise  to  the  level  of  triggering 
corrective  feedback.  A  data  collection  protocol  is  currently  under  IRB  review,  in  which 
military  attending  surgeons,  fellows  and  residents  will  perform  laceration  repair  and 
elementary  exercised  so  their  behavior  can  be  compared.  Ultimately,  this  will  form  a 
key  element  of  the  ocular  trauma  surgical  simulator  and  provide  new  tools  for 
developing  other  training  systems. 
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